{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"baselines-and-ceilings","title":"The number above it and the number below it","url":"https://addaly.com/learn/evaluating-ai/baselines-and-ceilings","position":9,"of":76,"minutes":9,"answer":"A score is only readable between a trivial baseline and the human agreement ceiling, and most impressive-looking results shrink to two or three points once both are drawn.","keyIdea":"A score is only readable between a trivial baseline and the human agreement ceiling, and most impressive-looking results shrink to two or three points once both are drawn."},"body":{"format":"markdown","text":"## Seventy-eight per cent of what\n\nAn intent router is reported at 78% accuracy. Nobody in the room can tell whether that is good, and the argument that follows is about vibes.\n\nFour numbers make it readable, and all four are cheap.\n\n**The trivial baseline.** Always predict the most common class. If 61% of tickets are billing, the do-nothing system scores 61%. Your model's real contribution is 17 points, not 78.\n\n**The cheap baseline.** Twenty lines of keyword rules, or a bag-of-words logistic regression trained in a second on the same labels. On routing tasks this often lands at 72 to 76%. Now the language model's contribution is two to six points, at a per-call cost the regex does not have.\n\n**The incumbent.** Whatever the system replaces: the old model, or the human process. If agents currently route correctly 85% of the time, an 78% model is a regression dressed as a launch.\n\n**The ceiling.** Two people labelling the same items agree, say, 88% of the time. Above that, \"accuracy\" is mostly measuring which annotator you happened to score against.\n\nWritten out, the result is not \"78%\". It is: *trivial 61, keywords 74, model 78, humans 88*. That sentence takes ten seconds to read and settles the decision.\n\n> Figure: Seventy-eight per cent of what. Bars (% accuracy on the same set): Trivial: always say 'billing' 61; Cheap: twenty keyword rules 74; Hosted language model 78; Ceiling: two humans agree 88. The model adds 17 points over the trivial baseline and four over keyword rules, bought with 900 ms of latency and a vendor in the failure path. Above 88 the labels themselves disagree.\n\n## Building them properly\n\nFree tools do all of this. `scikit-learn` has the trivial baselines built in:\n\n```python\nfrom sklearn.dummy import DummyClassifier\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.pipeline import make_pipeline\n\nDummyClassifier(strategy=\"most_frequent\").fit(X, y).score(Xte, yte)   # 0.61\nmake_pipeline(TfidfVectorizer(), LogisticRegression(max_iter=1000)\n              ).fit(X, y).score(Xte, yte)                             # 0.74\n```\n\nFor retrieval, the equivalent is BM25: keyword scoring that has been the strong classical baseline for thirty years. The `rank_bm25` package runs it on a laptop with no GPU. On corpora full of product codes, part numbers and jargon it frequently comes within a few points of an embedding model, for the mechanical reason that exact term matching is unbeatable on rare exact terms, which is precisely what a vector average smooths away. Teams that skip this baseline spend a quarter building a vector database to gain three points they could have measured in an afternoon.\n\nSpend an hour on the baseline, not five minutes. A badly built baseline flatters you, and everyone downstream inherits the flattery. Write down its configuration next to its score, the same way you would for the model.\n\n## The ceiling is not 100%\n\nA great deal of wasted effort comes from treating perfect as the target on tasks where the labels themselves are contested. If two careful annotators agree at 88%, then:\n\n- A model at 86% is roughly at the level of a second human. Further gains are partly fitting one annotator's idiosyncrasies.\n- The way to move past it is not a better model. It is a better definition: sharper rules, split categories, or a task decomposed into questions people can agree on.\n\nThe previous lesson's kappa number is what makes this concrete. Compute it once, and it stops the \"why are we not at 95%\" conversation permanently.\n\n## Baselines have costs, and that is part of the comparison\n\nReport cost and latency for the baseline too, because that is where the argument actually ends. A realistic line for a routing feature:\n\n- keyword rules: 74%, 0.4 ms, no per-call cost, no external dependency\n- small local model: 76%, 40 ms, no per-call cost, runs on a CPU\n- hosted large model: 78%, 900 ms, 0.2 cents per call, a vendor in the failure path\n\nFour points cost you nearly a second of latency and a dependency that can change under you. Sometimes worth it: an assistant that answers ambiguous free text often is. Sometimes plainly not. The point is that the decision becomes visible instead of being made by whoever spoke last.\n\n## The baseline nobody writes down\n\nFor a product, there is one more: **not having the feature at all**. Some AI features are measured against a fantasy of what users wanted rather than against the plain link, the search box, or the form that already worked. If a summary of a document is right 82% of the time and the document is three paragraphs long, the honest comparison is with reading the document.\n\nAsk it explicitly, before any of the arithmetic: what happens if we ship nothing here. Then measure against that."},"check":{"question":"A classifier for a rare compliance breach reports 97% accuracy on a set where 4% of items are breaches. Which single figure most changes how you read it?","options":[{"text":"The size of the test set, since 97% on 100 items has an error bar of roughly eight points.","correct":false,"tempting":"The interval matters, but it does not change the interpretation here: even the top of the interval is beaten by a system that never flags anything."},{"text":"The score of a model that always predicts \"not a breach\", which is 96% on this class balance.","correct":true},{"text":"The F1 score, which combines precision and recall into one comparable number.","correct":false,"tempting":"An improvement on accuracy, but it is still a score with nothing to compare it against; without the trivial baseline you cannot say whether any of the model's behaviour is doing work."},{"text":"The inter-annotator agreement, since that sets the ceiling the model is being measured against.","correct":false,"tempting":"The ceiling matters for interpreting near-ceiling results; this result is not near the ceiling, it is barely above the floor, and the floor is the figure that reframes it."}]},"prev":{"slug":"splits-and-leakage","title":"Two sets, and the ways they get contaminated","url":"https://addaly.com/learn/evaluating-ai/splits-and-leakage"},"next":{"slug":"error-bars-you-can-compute","title":"How wrong your number probably is","url":"https://addaly.com/learn/evaluating-ai/error-bars-you-can-compute"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The number above it and the number below it', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/baselines-and-ceilings (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The number above it and the number below it","courseTitle":"Knowing If It Works","module":{"index":1,"slug":"evidence","title":"What counts as evidence"},"answer":null,"keyIdea":"A score is only readable between a trivial baseline and the human agreement ceiling, and most impressive-looking results shrink to two or three points once both are drawn.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% accuracy on the same set","title":"Seventy-eight per cent of what","series":[{"label":"Trivial: always say ‘billing’","value":61},{"label":"Cheap: twenty keyword rules","value":74},{"label":"Hosted language model","value":78,"highlight":true},{"label":"Ceiling: two humans agree","value":88}],"caption":"The model adds 17 points over the trivial baseline and four over keyword rules, bought with 900 ms of latency and a vendor in the failure path. Above 88 the labels themselves disagree."},"alt":"Figure: Seventy-eight per cent of what. Bars (% accuracy on the same set): Trivial: always say ‘billing’ 61; Cheap: twenty keyword rules 74; Hosted language model 78; Ceiling: two humans agree 88. The model adds 17 points over the trivial baseline and four over keyword rules, bought with 900 ms of latency and a vendor in the failure path. Above 88 the labels themselves disagree."}},"blocks":[{"t":"h2","id":"seventy-eight-per-cent-of-what","runs":[{"text":"Seventy-eight per cent of what"}]},{"t":"p","runs":[{"text":"An intent router is reported at 78% accuracy. Nobody in the room can tell whether that is good, and the argument that follows is about vibes."}]},{"t":"p","runs":[{"text":"Four numbers make it readable, and all four are cheap."}]},{"t":"p","runs":[{"text":"The trivial baseline.","b":true},{"text":" Always predict the most common class. If 61% of tickets are billing, the do-nothing system scores 61%. Your model’s real contribution is 17 points, not 78."}]},{"t":"p","runs":[{"text":"The cheap baseline.","b":true},{"text":" Twenty lines of keyword rules, or a bag-of-words logistic regression trained in a second on the same labels. On routing tasks this often lands at 72 to 76%. Now the language model’s contribution is two to six points, at a per-call cost the regex does not have."}]},{"t":"p","runs":[{"text":"The incumbent.","b":true},{"text":" Whatever the system replaces: the old model, or the human process. If agents currently route correctly 85% of the time, an 78% model is a regression dressed as a launch."}]},{"t":"p","runs":[{"text":"The ceiling.","b":true},{"text":" Two people labelling the same items agree, say, 88% of the time. Above that, “accuracy” is mostly measuring which annotator you happened to score against."}]},{"t":"p","runs":[{"text":"Written out, the result is not “78%”. It is: "},{"text":"trivial 61, keywords 74, model 78, humans 88","i":true},{"text":". That sentence takes ten seconds to read and settles the decision."}]},{"t":"h2","id":"building-them-properly","runs":[{"text":"Building them properly"}]},{"t":"p","runs":[{"text":"Free tools do all of this. "},{"text":"scikit-learn","code":true},{"text":" has the trivial baselines built in:"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.dummy import DummyClassifier\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.pipeline import make_pipeline\n\nDummyClassifier(strategy=\"most_frequent\").fit(X, y).score(Xte, yte)   # 0.61\nmake_pipeline(TfidfVectorizer(), LogisticRegression(max_iter=1000)\n              ).fit(X, y).score(Xte, yte)                             # 0.74"},{"t":"p","runs":[{"text":"For retrieval, the equivalent is BM25: keyword scoring that has been the strong classical baseline for thirty years. The "},{"text":"rank_bm25","code":true},{"text":" package runs it on a laptop with no GPU. On corpora full of product codes, part numbers and jargon it frequently comes within a few points of an embedding model, for the mechanical reason that exact term matching is unbeatable on rare exact terms, which is precisely what a vector average smooths away. Teams that skip this baseline spend a quarter building a vector database to gain three points they could have measured in an afternoon."}]},{"t":"p","runs":[{"text":"Spend an hour on the baseline, not five minutes. A badly built baseline flatters you, and everyone downstream inherits the flattery. Write down its configuration next to its score, the same way you would for the model."}]},{"t":"h2","id":"the-ceiling-is-not-100","runs":[{"text":"The ceiling is not 100%"}]},{"t":"p","runs":[{"text":"A great deal of wasted effort comes from treating perfect as the target on tasks where the labels themselves are contested. If two careful annotators agree at 88%, then:"}]},{"t":"ul","items":[[{"text":"A model at 86% is roughly at the level of a second human. Further gains are partly fitting one annotator’s idiosyncrasies."}],[{"text":"The way to move past it is not a better model. It is a better definition: sharper rules, split categories, or a task decomposed into questions people can agree on."}]]},{"t":"p","runs":[{"text":"The previous lesson’s kappa number is what makes this concrete. Compute it once, and it stops the “why are we not at 95%” conversation permanently."}]},{"t":"h2","id":"baselines-have-costs-and-that-is-part-of-the-comparison","runs":[{"text":"Baselines have costs, and that is part of the comparison"}]},{"t":"p","runs":[{"text":"Report cost and latency for the baseline too, because that is where the argument actually ends. A realistic line for a routing feature:"}]},{"t":"ul","items":[[{"text":"keyword rules: 74%, 0.4 ms, no per-call cost, no external dependency"}],[{"text":"small local model: 76%, 40 ms, no per-call cost, runs on a CPU"}],[{"text":"hosted large model: 78%, 900 ms, 0.2 cents per call, a vendor in the failure path"}]]},{"t":"p","runs":[{"text":"Four points cost you nearly a second of latency and a dependency that can change under you. Sometimes worth it: an assistant that answers ambiguous free text often is. Sometimes plainly not. The point is that the decision becomes visible instead of being made by whoever spoke last."}]},{"t":"h2","id":"the-baseline-nobody-writes-down","runs":[{"text":"The baseline nobody writes down"}]},{"t":"p","runs":[{"text":"For a product, there is one more: "},{"text":"not having the feature at all","b":true},{"text":". Some AI features are measured against a fantasy of what users wanted rather than against the plain link, the search box, or the form that already worked. If a summary of a document is right 82% of the time and the document is three paragraphs long, the honest comparison is with reading the document."}]},{"t":"p","runs":[{"text":"Ask it explicitly, before any of the arithmetic: what happens if we ship nothing here. Then measure against that."}]}],"check":{"question":[{"text":"A classifier for a rare compliance breach reports 97% accuracy on a set where 4% of items are breaches. Which single figure most changes how you read it?"}],"options":[{"text":[{"text":"The size of the test set, since 97% on 100 items has an error bar of roughly eight points."}],"correct":false,"tempting":[{"text":"The interval matters, but it does not change the interpretation here: even the top of the interval is beaten by a system that never flags anything."}]},{"text":[{"text":"The score of a model that always predicts “not a breach”, which is 96% on this class balance."}],"correct":true,"tempting":null},{"text":[{"text":"The F1 score, which combines precision and recall into one comparable number."}],"correct":false,"tempting":[{"text":"An improvement on accuracy, but it is still a score with nothing to compare it against; without the trivial baseline you cannot say whether any of the model’s behaviour is doing work."}]},{"text":[{"text":"The inter-annotator agreement, since that sets the ceiling the model is being measured against."}],"correct":false,"tempting":[{"text":"The ceiling matters for interpreting near-ceiling results; this result is not near the ceiling, it is barely above the floor, and the floor is the figure that reframes it."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"splits-and-leakage","title":"Two sets, and the ways they get contaminated","minutes":9},"next":{"slug":"error-bars-you-can-compute","title":"How wrong your number probably is","minutes":9}}}