Batch, online, and the latency budget

Three shapes of serving

Batch. Score everything on a schedule, write the results to a table, and let consumers read them. A nightly job computes tomorrow’s churn risk for six million customers.

Simple, cheap, easy to monitor, and easy to roll back: you can rerun yesterday. Its limitation is freshness: a customer who signed up this morning has no score until tonight, and a customer whose circumstances changed at 10am is scored on last night’s picture.

Online. A service receives a request and returns a prediction. Fraud scoring, search ranking, anything reacting to a user action.

Fresh, and it drags in everything hard: latency budgets, availability, autoscaling, and the requirement that every feature be computable within the request.

Streaming. Predictions computed continuously as events arrive, results written where consumers can read them. A middle ground for systems built on Kafka or similar.

Most teams need less online serving than they assume. Ask what the freshness requirement actually is: often “within an hour” is fine, and an hourly batch job is dramatically cheaper to build and operate than a service.

The latency budget

If you serve online, somebody has a number. Say the API must answer in 100ms at the 99th percentile. That budget covers everything:

network in/out          15 ms
feature lookup          40 ms      <- usually the largest
preprocessing            5 ms
model inference         12 ms
response assembly        8 ms
headroom                20 ms

Notice where the time goes. It is rarely the model. Feature retrieval is usually the bottleneck: a database round trip for the customer’s 30-day transaction count costs more than a boosted tree traversal.

That has a design consequence. Precompute what you can. Features that change slowly (account age band, historical averages, segment) can be computed nightly and cached. Only real-time features are computed in the request. Doing this well is most of what makes an online model affordable.

And measure the tail, not the mean. A mean of 40ms with a p99 of 900ms is a bad service, because 1% of your users are having a terrible time and they are disproportionately your heaviest ones.

Making inference faster

In rough order of effort:

• Batch requests together. Ten predictions in one call is far cheaper than ten calls, because the per-call overhead dominates for small models.
• Use fewer, deeper trees. 200 trees of depth 8 usually serve faster than 1,000 of depth 4 at similar accuracy.
• Compile the model. Treelite and ONNX Runtime convert a model into optimised code, often 2 to 10 times faster with identical predictions.
• Cache. If the same inputs recur (the same user, the same query) cache the result with a short expiry.
• Use a smaller model. The next lesson but two.

Where a request goes wrong

The failure modes worth handling before launch, because each one has bitten somebody:

A missing feature. The feature store is slow or down. Decide now: fail the request, or predict with an imputed value and flag it. Both are defensible; neither should be discovered at 3am.

An unseen category. Module 3’s handle_unknown="ignore". Without it, the transform raises and the request 500s.

An out-of-range value. A negative age, an income of 10^12. Validate inputs at the boundary and reject or clip explicitly rather than letting the model extrapolate silently.

The model file will not load. Version mismatch between the training environment and the serving one. Pin versions, and check that the artefact loads as part of your deploy, before traffic reaches it.

Shadow and canary

Two deployment patterns that cost little and prevent most incidents.

Shadow mode. Run the new model alongside the old one, log both predictions, serve only the old one. After a week you know how the new model behaves on real traffic (including on the inputs your test set never contained) with no risk at all. This is the single most useful practice in this lesson.

Canary. Route 1% of traffic to the new model, watch the metrics, increase gradually. Standard software practice, and it applies unchanged.

Both require that you can run two models at once and route between them, which is worth building early because everything else depends on it.