{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"the-classifier-router","title":"Pattern: one call, one label, then code","url":"https://addaly.com/learn/agents-and-automation/the-classifier-router","position":47,"of":72,"minutes":9,"answer":"Two people labelling the same hundred items set the ceiling for any classifier, so measure human agreement before you measure the model.","keyIdea":"Two people labelling the same hundred items set the ceiling for any classifier, so measure human agreement before you measure the model."},"body":{"format":"markdown","text":"## The shape\n\nAn item arrives. One model call returns a label from a fixed set. Your code routes on the label. That is the whole pattern, and it is the best value in this course.\n\nIt is cheap: one call, a short prompt, no loop, so the cost per item is a fraction of a penny and the latency is a second. It is testable: you can build a labelled set and get a confusion matrix. It is observable: the distribution of labels over time is a single chart that tells you when something has changed. And it fails gracefully: a wrongly routed ticket lands in the wrong queue and somebody moves it.\n\nIf a request can be met by a classifier, build the classifier. A remarkable number of \"we need an agent\" conversations end here.\n\n## The label definitions are the product\n\nNot the prompt, not the model. The definitions.\n\nWrite them with the people who currently do the work, and write the boundary cases down: *a request to change a delivery address is Account, not Delivery, unless the parcel has already shipped, in which case it is Delivery.* Every rule of that kind you fail to write is a rule the model will invent differently each week.\n\nThen do the thing almost nobody does. **Have two people label the same hundred items independently and measure how often they agree.** If they agree on 82 per cent, your model cannot meaningfully exceed 82 per cent, because there is no stable answer for the disagreement cases. You have discovered your ceiling before spending anything, and you have found the eighteen items whose definitions need rewriting.\n\nTeams who skip this spend months trying to push a model past a boundary that does not exist in the data. It is one afternoon and it changes the whole project's expectations.\n\n## Make \"unsure\" a label\n\nAdd `other` and `unsure` to the enum, with instructions for when to use them. Without an exit, every ambiguous item is forced into a category it does not belong in, and the wrongness is invisible because the output looks like every other output.\n\nThen set the routing policy from the cost asymmetry: confident labels route automatically, `unsure` goes to a person, and the proportion going to a person is a number you watch. If it is 40 per cent you have built a queue, not an automation.\n\n## Cheap layers first\n\nA keyword rule handles a surprising share of real traffic at zero cost and zero latency. Anything containing an order number in a known format is an order enquiry. Anything from the payments provider's domain is a payment notification.\n\nRun the rules first, send the remainder to the model. At ten thousand items a day, moving 40 per cent to rules cuts the bill by 40 per cent and removes those items from the set that can be misclassified at all.\n\n## Classic machine learning wins once you have labels\n\nThis is worth saying plainly because the current fashion obscures it. With five hundred labelled examples and a narrow set of classes, a logistic regression over TF-IDF features or over embeddings frequently matches a language model on classification, runs in a millisecond, costs nothing per call, and is completely deterministic.\n\n```python\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.pipeline import make_pipeline\n\nclf = make_pipeline(TfidfVectorizer(ngram_range=(1,2)), LogisticRegression(max_iter=1000))\nclf.fit(train_texts, train_labels)\n```\n\nScikit-learn is free and runs on any laptop. The honest comparison: the language model wins when you have no labelled data, when the classes change often, or when the decision needs world knowledge. The classic model wins on cost, latency and stability once labels exist, and the labels arrive for free if humans are correcting the queue anyway.\n\nMany good systems use both: the model labels new items, humans correct, and the corrections train a cheap classifier that gradually takes over the routine traffic.\n\n> Figure: The cheapest drift detector there is. Line chart of Share of items, % against Week: Labelled other (1, 4) (2, 4) (3, 5) (4, 4) (5, 6) (6, 11) (7, 18) (8, 23); Labelled delivery (1, 31) (2, 30) (3, 32) (4, 30) (5, 29) (6, 25) (7, 20) (8, 16). A new product shipped in week five. Nobody labelled anything and no accuracy was measured: the share of items falling into other simply rose, several weeks before the first complaint arrived.\n\n## Watch the distribution, not just the accuracy\n\nChart the share of each label per week. A sudden rise in `other` means the world changed (a new product, a new kind of complaint, a supplier who started sending a different document) and it shows up in the distribution weeks before it shows up in a complaint.\n\nThis is the cheapest drift detector in existence and it requires no labels at all, which is what makes it usable forever rather than for the first month."},"check":{"question":"Before building a ticket classifier, a team has two experienced agents independently label the same 100 tickets. They agree on 82. What does this number tell them?","options":[{"text":"The tickets are too varied for a fixed label set, so the categories should be replaced with free-text tagging.","correct":false,"tempting":"Abandons structure at the first sign of ambiguity, when the disagreements are usually concentrated in a few definable boundary cases."},{"text":"The agents need retraining, since consistent staff should agree on nearly every ticket.","correct":false,"tempting":"Blames the people for what is almost always missing definitions: the disagreements are evidence about the scheme, not about the raters."},{"text":"It sets a practical ceiling: with no stable answer on the disputed items, a model cannot reliably exceed that, and the 18 disagreements name the definitions to rewrite.","correct":true},{"text":"It gives a baseline to beat, since a model that agrees with both raters more than they agree with each other is performing above human level.","correct":false,"tempting":"There is no ground truth to be more right about on the contested items, so exceeding the agreement rate measures conformity to one rater, not accuracy."}]},"prev":{"slug":"disclosure-and-audit","title":"Telling people, and keeping the receipts","url":"https://addaly.com/learn/agents-and-automation/disclosure-and-audit"},"next":{"slug":"the-drafting-assistant","title":"Pattern: it writes, a person sends","url":"https://addaly.com/learn/agents-and-automation/the-drafting-assistant"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Pattern: one call, one label, then code', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/the-classifier-router (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Pattern: one call, one label, then code","courseTitle":"Agents and Automation","module":{"index":6,"slug":"automation-patterns","title":"Eight patterns that keep working"},"answer":null,"keyIdea":"Two people labelling the same hundred items set the ceiling for any classifier, so measure human agreement before you measure the model.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 6.1","figure":{"type":"line","title":"The cheapest drift detector there is","series":[{"label":"Labelled other","points":[[1,4],[2,4],[3,5],[4,4],[5,6],[6,11],[7,18],[8,23]]},{"label":"Labelled delivery","points":[[1,31],[2,30],[3,32],[4,30],[5,29],[6,25],[7,20],[8,16]]}],"xLabel":"Week","yLabel":"Share of items, %","caption":"A new product shipped in week five. Nobody labelled anything and no accuracy was measured: the share of items falling into other simply rose, several weeks before the first complaint arrived."},"alt":"Figure: The cheapest drift detector there is. Line chart of Share of items, % against Week: Labelled other (1, 4) (2, 4) (3, 5) (4, 4) (5, 6) (6, 11) (7, 18) (8, 23); Labelled delivery (1, 31) (2, 30) (3, 32) (4, 30) (5, 29) (6, 25) (7, 20) (8, 16). A new product shipped in week five. Nobody labelled anything and no accuracy was measured: the share of items falling into other simply rose, several weeks before the first complaint arrived."}},"blocks":[{"t":"h2","id":"the-shape","runs":[{"text":"The shape"}]},{"t":"p","runs":[{"text":"An item arrives. One model call returns a label from a fixed set. Your code routes on the label. That is the whole pattern, and it is the best value in this course."}]},{"t":"p","runs":[{"text":"It is cheap: one call, a short prompt, no loop, so the cost per item is a fraction of a penny and the latency is a second. It is testable: you can build a labelled set and get a confusion matrix. It is observable: the distribution of labels over time is a single chart that tells you when something has changed. And it fails gracefully: a wrongly routed ticket lands in the wrong queue and somebody moves it."}]},{"t":"p","runs":[{"text":"If a request can be met by a classifier, build the classifier. A remarkable number of “we need an agent” conversations end here."}]},{"t":"h2","id":"the-label-definitions-are-the-product","runs":[{"text":"The label definitions are the product"}]},{"t":"p","runs":[{"text":"Not the prompt, not the model. The definitions."}]},{"t":"p","runs":[{"text":"Write them with the people who currently do the work, and write the boundary cases down: "},{"text":"a request to change a delivery address is Account, not Delivery, unless the parcel has already shipped, in which case it is Delivery.","i":true},{"text":" Every rule of that kind you fail to write is a rule the model will invent differently each week."}]},{"t":"p","runs":[{"text":"Then do the thing almost nobody does. "},{"text":"Have two people label the same hundred items independently and measure how often they agree.","b":true},{"text":" If they agree on 82 per cent, your model cannot meaningfully exceed 82 per cent, because there is no stable answer for the disagreement cases. You have discovered your ceiling before spending anything, and you have found the eighteen items whose definitions need rewriting."}]},{"t":"p","runs":[{"text":"Teams who skip this spend months trying to push a model past a boundary that does not exist in the data. It is one afternoon and it changes the whole project’s expectations."}]},{"t":"h2","id":"make-unsure-a-label","runs":[{"text":"Make “unsure” a label"}]},{"t":"p","runs":[{"text":"Add "},{"text":"other","code":true},{"text":" and "},{"text":"unsure","code":true},{"text":" to the enum, with instructions for when to use them. Without an exit, every ambiguous item is forced into a category it does not belong in, and the wrongness is invisible because the output looks like every other output."}]},{"t":"p","runs":[{"text":"Then set the routing policy from the cost asymmetry: confident labels route automatically, "},{"text":"unsure","code":true},{"text":" goes to a person, and the proportion going to a person is a number you watch. If it is 40 per cent you have built a queue, not an automation."}]},{"t":"h2","id":"cheap-layers-first","runs":[{"text":"Cheap layers first"}]},{"t":"p","runs":[{"text":"A keyword rule handles a surprising share of real traffic at zero cost and zero latency. Anything containing an order number in a known format is an order enquiry. Anything from the payments provider’s domain is a payment notification."}]},{"t":"p","runs":[{"text":"Run the rules first, send the remainder to the model. At ten thousand items a day, moving 40 per cent to rules cuts the bill by 40 per cent and removes those items from the set that can be misclassified at all."}]},{"t":"h2","id":"classic-machine-learning-wins-once-you-have-labels","runs":[{"text":"Classic machine learning wins once you have labels"}]},{"t":"p","runs":[{"text":"This is worth saying plainly because the current fashion obscures it. With five hundred labelled examples and a narrow set of classes, a logistic regression over TF-IDF features or over embeddings frequently matches a language model on classification, runs in a millisecond, costs nothing per call, and is completely deterministic."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.linear_model import LogisticRegression\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.pipeline import make_pipeline\n\nclf = make_pipeline(TfidfVectorizer(ngram_range=(1,2)), LogisticRegression(max_iter=1000))\nclf.fit(train_texts, train_labels)"},{"t":"p","runs":[{"text":"Scikit-learn is free and runs on any laptop. The honest comparison: the language model wins when you have no labelled data, when the classes change often, or when the decision needs world knowledge. The classic model wins on cost, latency and stability once labels exist, and the labels arrive for free if humans are correcting the queue anyway."}]},{"t":"p","runs":[{"text":"Many good systems use both: the model labels new items, humans correct, and the corrections train a cheap classifier that gradually takes over the routine traffic."}]},{"t":"h2","id":"watch-the-distribution-not-just-the-accuracy","runs":[{"text":"Watch the distribution, not just the accuracy"}]},{"t":"p","runs":[{"text":"Chart the share of each label per week. A sudden rise in "},{"text":"other","code":true},{"text":" means the world changed (a new product, a new kind of complaint, a supplier who started sending a different document) and it shows up in the distribution weeks before it shows up in a complaint."}]},{"t":"p","runs":[{"text":"This is the cheapest drift detector in existence and it requires no labels at all, which is what makes it usable forever rather than for the first month."}]}],"check":{"question":[{"text":"Before building a ticket classifier, a team has two experienced agents independently label the same 100 tickets. They agree on 82. What does this number tell them?"}],"options":[{"text":[{"text":"The tickets are too varied for a fixed label set, so the categories should be replaced with free-text tagging."}],"correct":false,"tempting":[{"text":"Abandons structure at the first sign of ambiguity, when the disagreements are usually concentrated in a few definable boundary cases."}]},{"text":[{"text":"The agents need retraining, since consistent staff should agree on nearly every ticket."}],"correct":false,"tempting":[{"text":"Blames the people for what is almost always missing definitions: the disagreements are evidence about the scheme, not about the raters."}]},{"text":[{"text":"It sets a practical ceiling: with no stable answer on the disputed items, a model cannot reliably exceed that, and the 18 disagreements name the definitions to rewrite."}],"correct":true,"tempting":null},{"text":[{"text":"It gives a baseline to beat, since a model that agrees with both raters more than they agree with each other is performing above human level."}],"correct":false,"tempting":[{"text":"There is no ground truth to be more right about on the contested items, so exceeding the agreement rate measures conformity to one rater, not accuracy."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"disclosure-and-audit","title":"Telling people, and keeping the receipts","minutes":8},"next":{"slug":"the-drafting-assistant","title":"Pattern: it writes, a person sends","minutes":9}}}