The number above it and the number below it

Seventy-eight per cent of what

An intent router is reported at 78% accuracy. Nobody in the room can tell whether that is good, and the argument that follows is about vibes.

Four numbers make it readable, and all four are cheap.

The trivial baseline. Always predict the most common class. If 61% of tickets are billing, the do-nothing system scores 61%. Your model’s real contribution is 17 points, not 78.

The cheap baseline. Twenty lines of keyword rules, or a bag-of-words logistic regression trained in a second on the same labels. On routing tasks this often lands at 72 to 76%. Now the language model’s contribution is two to six points, at a per-call cost the regex does not have.

The incumbent. Whatever the system replaces: the old model, or the human process. If agents currently route correctly 85% of the time, an 78% model is a regression dressed as a launch.

The ceiling. Two people labelling the same items agree, say, 88% of the time. Above that, “accuracy” is mostly measuring which annotator you happened to score against.

Written out, the result is not “78%”. It is: trivial 61, keywords 74, model 78, humans 88. That sentence takes ten seconds to read and settles the decision.

Building them properly

Free tools do all of this. scikit-learn has the trivial baselines built in:

from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

DummyClassifier(strategy="most_frequent").fit(X, y).score(Xte, yte)   # 0.61
make_pipeline(TfidfVectorizer(), LogisticRegression(max_iter=1000)
              ).fit(X, y).score(Xte, yte)                             # 0.74

For retrieval, the equivalent is BM25: keyword scoring that has been the strong classical baseline for thirty years. The rank_bm25 package runs it on a laptop with no GPU. On corpora full of product codes, part numbers and jargon it frequently comes within a few points of an embedding model, for the mechanical reason that exact term matching is unbeatable on rare exact terms, which is precisely what a vector average smooths away. Teams that skip this baseline spend a quarter building a vector database to gain three points they could have measured in an afternoon.

Spend an hour on the baseline, not five minutes. A badly built baseline flatters you, and everyone downstream inherits the flattery. Write down its configuration next to its score, the same way you would for the model.

The ceiling is not 100%

A great deal of wasted effort comes from treating perfect as the target on tasks where the labels themselves are contested. If two careful annotators agree at 88%, then:

• A model at 86% is roughly at the level of a second human. Further gains are partly fitting one annotator’s idiosyncrasies.
• The way to move past it is not a better model. It is a better definition: sharper rules, split categories, or a task decomposed into questions people can agree on.

The previous lesson’s kappa number is what makes this concrete. Compute it once, and it stops the “why are we not at 95%” conversation permanently.

Baselines have costs, and that is part of the comparison

Report cost and latency for the baseline too, because that is where the argument actually ends. A realistic line for a routing feature:

• keyword rules: 74%, 0.4 ms, no per-call cost, no external dependency
• small local model: 76%, 40 ms, no per-call cost, runs on a CPU
• hosted large model: 78%, 900 ms, 0.2 cents per call, a vendor in the failure path

Four points cost you nearly a second of latency and a dependency that can change under you. Sometimes worth it: an assistant that answers ambiguous free text often is. Sometimes plainly not. The point is that the decision becomes visible instead of being made by whoever spoke last.

The baseline nobody writes down

For a product, there is one more: not having the feature at all. Some AI features are measured against a fantasy of what users wanted rather than against the plain link, the search box, or the form that already worked. If a summary of a document is right 82% of the time and the document is three paragraphs long, the honest comparison is with reading the document.

Ask it explicitly, before any of the arithmetic: what happens if we ship nothing here. Then measure against that.