Pattern: one call, one label, then code

The shape

An item arrives. One model call returns a label from a fixed set. Your code routes on the label. That is the whole pattern, and it is the best value in this course.

It is cheap: one call, a short prompt, no loop, so the cost per item is a fraction of a penny and the latency is a second. It is testable: you can build a labelled set and get a confusion matrix. It is observable: the distribution of labels over time is a single chart that tells you when something has changed. And it fails gracefully: a wrongly routed ticket lands in the wrong queue and somebody moves it.

If a request can be met by a classifier, build the classifier. A remarkable number of “we need an agent” conversations end here.

The label definitions are the product

Not the prompt, not the model. The definitions.

Write them with the people who currently do the work, and write the boundary cases down: a request to change a delivery address is Account, not Delivery, unless the parcel has already shipped, in which case it is Delivery. Every rule of that kind you fail to write is a rule the model will invent differently each week.

Then do the thing almost nobody does. Have two people label the same hundred items independently and measure how often they agree. If they agree on 82 per cent, your model cannot meaningfully exceed 82 per cent, because there is no stable answer for the disagreement cases. You have discovered your ceiling before spending anything, and you have found the eighteen items whose definitions need rewriting.

Teams who skip this spend months trying to push a model past a boundary that does not exist in the data. It is one afternoon and it changes the whole project’s expectations.

Make “unsure” a label

Add other and unsure to the enum, with instructions for when to use them. Without an exit, every ambiguous item is forced into a category it does not belong in, and the wrongness is invisible because the output looks like every other output.

Then set the routing policy from the cost asymmetry: confident labels route automatically, unsure goes to a person, and the proportion going to a person is a number you watch. If it is 40 per cent you have built a queue, not an automation.

Cheap layers first

A keyword rule handles a surprising share of real traffic at zero cost and zero latency. Anything containing an order number in a known format is an order enquiry. Anything from the payments provider’s domain is a payment notification.

Run the rules first, send the remainder to the model. At ten thousand items a day, moving 40 per cent to rules cuts the bill by 40 per cent and removes those items from the set that can be misclassified at all.

Classic machine learning wins once you have labels

This is worth saying plainly because the current fashion obscures it. With five hundred labelled examples and a narrow set of classes, a logistic regression over TF-IDF features or over embeddings frequently matches a language model on classification, runs in a millisecond, costs nothing per call, and is completely deterministic.

from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import make_pipeline

clf = make_pipeline(TfidfVectorizer(ngram_range=(1,2)), LogisticRegression(max_iter=1000))
clf.fit(train_texts, train_labels)

Scikit-learn is free and runs on any laptop. The honest comparison: the language model wins when you have no labelled data, when the classes change often, or when the decision needs world knowledge. The classic model wins on cost, latency and stability once labels exist, and the labels arrive for free if humans are correcting the queue anyway.

Many good systems use both: the model labels new items, humans correct, and the corrections train a cheap classifier that gradually takes over the routine traffic.

Watch the distribution, not just the accuracy

Chart the share of each label per week. A sudden rise in other means the world changed (a new product, a new kind of complaint, a supplier who started sending a different document) and it shows up in the distribution weeks before it shows up in a complaint.

This is the cheapest drift detector in existence and it requires no labels at all, which is what makes it usable forever rather than for the first month.