Spending money where it matters

The shape production systems actually have

People assume the systems they admire run the best available model on every request. Very few do. The common design is a cascade: a cheap model handles everything, and only the cases it cannot settle go to an expensive one.

1,000 items
  -> small model handles all 1,000            $0.40
  -> 120 flagged uncertain go to large model  $2.60
                                              -----
                                              $3.00
  versus all 1,000 on the large model:       $21.60

Eighty-six per cent saved, with the hard cases still getting the expensive treatment. This is not a compromise design. On many tasks it also scores better than the large model alone, because the escalation step forces a second look at exactly the items most likely to be wrong.

Deciding what escalates

You need a signal for “this one is difficult”, and there are four that work without any special infrastructure.

Ask for a confidence word. Have the small model end with CONFIDENT or UNSURE. It is not calibrated and it is not a probability, and it still correlates usefully with correctness: well enough to route on. Check the correlation on fifty items before trusting it.

Rules you already know. Some cases are structurally hard: a document over thirty pages, a ticket from a top-tier customer, an invoice above a threshold, a message in a language the small model handles poorly. Route those up without asking anything.

Disagreement. Run the small model twice at a non-zero temperature. If the two answers differ, escalate. Cheap, crude, and effective on classification.

Cost of being wrong. The most important one, and it is a business decision rather than a technical one. Anything where a mistake costs money, safety or a customer goes to the expensive model regardless of how confident anything is.

The measurement that makes it honest

A cascade is only correct if the small model is adequate on the easy cases, and the only way to know is to check.

Take 100 items. Run both models. Compare. You will get four groups, and the interesting one is the fourth:

• Both right: the cascade is safe here.
• Both wrong: the model is not the problem; the prompt or the task is.
• Large right, small wrong: this is what you are paying the large model for. How many, and do your escalation rules catch them?
• Small right, large wrong. This group is always bigger than people expect. Large models are not uniformly better; they are better on average, and they overthink simple cases.

If your escalation rules catch most of the third group, the cascade is working. If they do not, tighten the rules rather than abandoning the design.

Where cascades are the wrong idea

When latency matters more than money. Two model calls take twice as long. For anything a person is waiting on, that can be the wrong trade.

When the volume is small. A hundred items a day at a cent each is a dollar. Engineering a cascade to save 80% of a dollar is not a good use of an afternoon.

When you cannot measure. A cascade without the 100-item comparison is a guess that quietly ships worse work. If you are not going to check, use the model you trust.

The cheapest escalation of all

There is a version of the third group that costs nothing to catch: you. In work where a person is reading the output anyway (drafting, correspondence, review queues) the escalation rule can be that you notice. Run everything on the cheap model, and when an answer looks thin, ask the expensive one the same question.

That is a cascade with a human as the confidence signal, and on small volumes it beats any automated version, because you are far better calibrated about your own work than a model reporting on its own certainty.

The version for a person, not a system

You do not need code for any of this. The habit is the same:

• Default to the cheap, fast model for everything routine: rewriting, summarising, formatting, ordinary questions.
• Move to the strong model when the answer matters, when the question is hard, or when the cheap one has given you something you doubt.
• Notice when you last used the expensive model out of habit rather than need.

People who work this way spend a fraction of what people who leave the strongest model selected all day spend, get answers faster for most of their work, and do not lose anything. The cheap model is not a compromise for two thirds of what anybody actually does.