Measuring the cache instead of believing in it

The number is in the response

Every provider that offers prefix caching reports, on each response, how many input tokens were read from cache and how many were not. Those two numbers give you the only cache metric that matters:

hit_rate = cached_in / (cached_in + uncached_in)

Log it per request, alongside the assembly version and the request type. Everything else in this lesson is what you do with that series.

Teams routinely ship a carefully ordered prefix and never check whether it hits. A surprising share of the time it does not, for one of the reasons in the previous lesson, and nothing in the system says so: the requests succeed, the answers are fine, the bill is quietly 60% higher than the design predicted.

What good looks like

There is no universal target, because it depends entirely on your traffic. What you can do is compute the expected rate from your own layout and compare.

If your stable block is 1,400 tokens and a typical request is 9,000 tokens, the ceiling is about 16% on the first turn: the stable block over the whole request. In a conversation with history caching, it climbs with each turn, because history is append-only: by turn eight, most of the request is a prefix that existed on turn seven.

So do not compare against a number from somebody else’s system. Compute your own ceiling and watch the gap.

Three alerts worth having

A drop after a deploy. The most common cause of a sudden cost rise, and the easiest to catch. Alert on a day-over-day fall of more than a few points in the median hit rate. It usually means someone introduced a variable element above the boundary, and you find out in an hour rather than on the invoice.

A rate near zero on a specific request type. Averages hide this. Slice by feature: one endpoint interpolating a user id into its system prompt will sit near zero while the fleet average looks acceptable.

A rising write-to-read ratio. If you are writing far more cache entries than you read, entries are expiring before reuse. Either traffic is too sparse for caching to pay, or your prefix is more variable than you think, and the break-even calculation from the last lesson says which side of the line you are on.

Cold starts are real and brief

Every deploy that changes the stable block starts an empty cache. For a few minutes, every request pays the write premium and none read.

This is fine and worth knowing so you do not misread the graph. Two practical notes. Deploying during peak traffic means paying the write premium on a large number of concurrent requests, several of which will write the same entry before any of them can read it: deploy off-peak where you can. And a rolling deploy with two prompt versions live at once maintains two cache entries, which is normal and briefly doubles the write cost.

Do not tune the prompt for the cache

One discipline worth stating, because it is a trap that catches good engineers.

The cache rewards stability, so there is a temptation to freeze the prompt, refuse improvements, or move useful variable context out of the request to protect the hit rate. That is optimising the metric rather than the product.

The cache is worth roughly a 90% discount on the tokens it covers. A change that improves accuracy by two points is almost always worth more than the cache entry it invalidates, because the accuracy shows up in cost per resolution and the cache shows up in cost per call. Make the change, accept a few minutes of cold cache, and move on.

Verify the behaviour, do not trust the documentation

Providers differ in the details: minimum lengths, whether images and tool definitions participate, how breakpoints work, how long entries live, whether the cache is shared across your organisation’s keys. The documentation is usually right and occasionally out of date, and the behaviour changes between model versions.

So verify with a two-request experiment whenever you adopt caching on a new model:

a = call(msgs)        # cold
b = call(msgs)        # identical, immediately after
print(a.usage, b.usage)

If b shows the cached count you expected, the mechanism works as you think. If it does not, you have just saved yourself a month of assuming a discount you were never receiving. This is a five-minute check and it belongs in the runbook for every model upgrade.