{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"serving-a-model","title":"Batch, online, and the latency budget","url":"https://addaly.com/learn/machine-learning-foundations/serving-a-model","position":84,"of":92,"minutes":8,"answer":"Feature retrieval usually dominates the latency budget rather than model inference, so precompute slow-moving features, and run a new model in shadow mode against real traffic before it serves anyone.","keyIdea":"Feature retrieval usually dominates the latency budget rather than model inference, so precompute slow-moving features, and run a new model in shadow mode against real traffic before it serves anyone."},"body":{"format":"markdown","text":"## Three shapes of serving\n\n**Batch.** Score everything on a schedule, write the results to a table, and let consumers read them. A nightly job computes tomorrow's churn risk for six million customers.\n\nSimple, cheap, easy to monitor, and easy to roll back: you can rerun yesterday. Its limitation is freshness: a customer who signed up this morning has no score until tonight, and a customer whose circumstances changed at 10am is scored on last night's picture.\n\n**Online.** A service receives a request and returns a prediction. Fraud scoring, search ranking, anything reacting to a user action.\n\nFresh, and it drags in everything hard: latency budgets, availability, autoscaling, and the requirement that every feature be computable within the request.\n\n**Streaming.** Predictions computed continuously as events arrive, results written where consumers can read them. A middle ground for systems built on Kafka or similar.\n\nMost teams need less online serving than they assume. Ask what the freshness requirement actually is: often \"within an hour\" is fine, and an hourly batch job is dramatically cheaper to build and operate than a service.\n\n## The latency budget\n\nIf you serve online, somebody has a number. Say the API must answer in 100ms at the 99th percentile. That budget covers everything:\n\n```text\nnetwork in/out          15 ms\nfeature lookup          40 ms      <- usually the largest\npreprocessing            5 ms\nmodel inference         12 ms\nresponse assembly        8 ms\nheadroom                20 ms\n```\n\nNotice where the time goes. It is rarely the model. **Feature retrieval is usually the bottleneck**: a database round trip for the customer's 30-day transaction count costs more than a boosted tree traversal.\n\n> Figure: A 100 millisecond budget at the 99th percentile. Bars (ms): Network in and out 15; Feature lookup 40; Preprocessing 5; Model inference 12; Response assembly 8; Headroom 20. The model is 12 of the 100 milliseconds. The database round trip for a customer's 30-day transaction count costs more than a boosted tree traversal, which is why slow-moving features are computed nightly and cached, and only the genuinely real-time ones are fetched inside the request.\n\nThat has a design consequence. Precompute what you can. Features that change slowly (account age band, historical averages, segment) can be computed nightly and cached. Only real-time features are computed in the request. Doing this well is most of what makes an online model affordable.\n\nAnd measure the tail, not the mean. A mean of 40ms with a p99 of 900ms is a bad service, because 1% of your users are having a terrible time and they are disproportionately your heaviest ones.\n\n## Making inference faster\n\nIn rough order of effort:\n\n- **Batch requests together.** Ten predictions in one call is far cheaper than ten calls, because the per-call overhead dominates for small models.\n- **Use fewer, deeper trees.** 200 trees of depth 8 usually serve faster than 1,000 of depth 4 at similar accuracy.\n- **Compile the model.** Treelite and ONNX Runtime convert a model into optimised code, often 2 to 10 times faster with identical predictions.\n- **Cache.** If the same inputs recur (the same user, the same query) cache the result with a short expiry.\n- **Use a smaller model.** The next lesson but two.\n\n## Where a request goes wrong\n\nThe failure modes worth handling before launch, because each one has bitten somebody:\n\n**A missing feature.** The feature store is slow or down. Decide now: fail the request, or predict with an imputed value and flag it. Both are defensible; neither should be discovered at 3am.\n\n**An unseen category.** Module 3's `handle_unknown=\"ignore\"`. Without it, the transform raises and the request 500s.\n\n**An out-of-range value.** A negative age, an income of 10^12. Validate inputs at the boundary and reject or clip explicitly rather than letting the model extrapolate silently.\n\n**The model file will not load.** Version mismatch between the training environment and the serving one. Pin versions, and check that the artefact loads as part of your deploy, before traffic reaches it.\n\n## Shadow and canary\n\nTwo deployment patterns that cost little and prevent most incidents.\n\n**Shadow mode.** Run the new model alongside the old one, log both predictions, serve only the old one. After a week you know how the new model behaves on real traffic (including on the inputs your test set never contained) with no risk at all. This is the single most useful practice in this lesson.\n\n**Canary.** Route 1% of traffic to the new model, watch the metrics, increase gradually. Standard software practice, and it applies unchanged.\n\nBoth require that you can run two models at once and route between them, which is worth building early because everything else depends on it."},"check":{"question":"An online fraud service has a mean latency of 35ms and a p99 of 850ms against a 100ms budget. Where should the team look first?","options":[{"text":"The model's inference time, since the tail suggests some predictions require deeper tree traversal.","correct":false,"tempting":"Tree traversal time varies little between inputs, so it cannot produce a twenty-fold spread between the mean and the tail."},{"text":"The mean, since 35ms is already a third of the budget and reducing it lowers the tail proportionally.","correct":false,"tempting":"Tail latency usually has a different cause from the mean, so improving the mean rarely moves the p99 proportionally."},{"text":"Feature retrieval, since a database round trip that occasionally misses a cache or waits on a slow query is the usual source of a long tail while the mean stays low.","correct":true},{"text":"Network time, since 1% of requests may be coming from distant regions.","correct":false,"tempting":"Geographic variation does add tail latency, but it is bounded and typically far smaller than the 800ms excess described here."}]},"prev":{"slug":"the-model-is-not-the-system","title":"The model is about 5% of what you have to build","url":"https://addaly.com/learn/machine-learning-foundations/the-model-is-not-the-system"},"next":{"slug":"training-serving-skew","title":"When training and serving quietly disagree","url":"https://addaly.com/learn/machine-learning-foundations/training-serving-skew"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Batch, online, and the latency budget', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/serving-a-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Batch, online, and the latency budget","courseTitle":"Machine Learning, Foundations","module":{"index":9,"slug":"into-the-world","title":"Into the world, and keeping it honest"},"answer":null,"keyIdea":"Feature retrieval usually dominates the latency budget rather than model inference, so precompute slow-moving features, and run a new model in shadow mode against real traffic before it serves anyone.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"ms","title":"A 100 millisecond budget at the 99th percentile","series":[{"label":"Network in and out","value":15},{"label":"Feature lookup","value":40,"highlight":true},{"label":"Preprocessing","value":5},{"label":"Model inference","value":12},{"label":"Response assembly","value":8},{"label":"Headroom","value":20}],"caption":"The model is 12 of the 100 milliseconds. The database round trip for a customer’s 30-day transaction count costs more than a boosted tree traversal, which is why slow-moving features are computed nightly and cached, and only the genuinely real-time ones are fetched inside the request."},"alt":"Figure: A 100 millisecond budget at the 99th percentile. Bars (ms): Network in and out 15; Feature lookup 40; Preprocessing 5; Model inference 12; Response assembly 8; Headroom 20. The model is 12 of the 100 milliseconds. The database round trip for a customer’s 30-day transaction count costs more than a boosted tree traversal, which is why slow-moving features are computed nightly and cached, and only the genuinely real-time ones are fetched inside the request."}},"blocks":[{"t":"h2","id":"three-shapes-of-serving","runs":[{"text":"Three shapes of serving"}]},{"t":"p","runs":[{"text":"Batch.","b":true},{"text":" Score everything on a schedule, write the results to a table, and let consumers read them. A nightly job computes tomorrow’s churn risk for six million customers."}]},{"t":"p","runs":[{"text":"Simple, cheap, easy to monitor, and easy to roll back: you can rerun yesterday. Its limitation is freshness: a customer who signed up this morning has no score until tonight, and a customer whose circumstances changed at 10am is scored on last night’s picture."}]},{"t":"p","runs":[{"text":"Online.","b":true},{"text":" A service receives a request and returns a prediction. Fraud scoring, search ranking, anything reacting to a user action."}]},{"t":"p","runs":[{"text":"Fresh, and it drags in everything hard: latency budgets, availability, autoscaling, and the requirement that every feature be computable within the request."}]},{"t":"p","runs":[{"text":"Streaming.","b":true},{"text":" Predictions computed continuously as events arrive, results written where consumers can read them. A middle ground for systems built on Kafka or similar."}]},{"t":"p","runs":[{"text":"Most teams need less online serving than they assume. Ask what the freshness requirement actually is: often “within an hour” is fine, and an hourly batch job is dramatically cheaper to build and operate than a service."}]},{"t":"h2","id":"the-latency-budget","runs":[{"text":"The latency budget"}]},{"t":"p","runs":[{"text":"If you serve online, somebody has a number. Say the API must answer in 100ms at the 99th percentile. That budget covers everything:"}]},{"t":"code","lang":"text","label":"Text","text":"network in/out          15 ms\nfeature lookup          40 ms      <- usually the largest\npreprocessing            5 ms\nmodel inference         12 ms\nresponse assembly        8 ms\nheadroom                20 ms"},{"t":"p","runs":[{"text":"Notice where the time goes. It is rarely the model. "},{"text":"Feature retrieval is usually the bottleneck","b":true},{"text":": a database round trip for the customer’s 30-day transaction count costs more than a boosted tree traversal."}]},{"t":"p","runs":[{"text":"That has a design consequence. Precompute what you can. Features that change slowly (account age band, historical averages, segment) can be computed nightly and cached. Only real-time features are computed in the request. Doing this well is most of what makes an online model affordable."}]},{"t":"p","runs":[{"text":"And measure the tail, not the mean. A mean of 40ms with a p99 of 900ms is a bad service, because 1% of your users are having a terrible time and they are disproportionately your heaviest ones."}]},{"t":"h2","id":"making-inference-faster","runs":[{"text":"Making inference faster"}]},{"t":"p","runs":[{"text":"In rough order of effort:"}]},{"t":"ul","items":[[{"text":"Batch requests together.","b":true},{"text":" Ten predictions in one call is far cheaper than ten calls, because the per-call overhead dominates for small models."}],[{"text":"Use fewer, deeper trees.","b":true},{"text":" 200 trees of depth 8 usually serve faster than 1,000 of depth 4 at similar accuracy."}],[{"text":"Compile the model.","b":true},{"text":" Treelite and ONNX Runtime convert a model into optimised code, often 2 to 10 times faster with identical predictions."}],[{"text":"Cache.","b":true},{"text":" If the same inputs recur (the same user, the same query) cache the result with a short expiry."}],[{"text":"Use a smaller model.","b":true},{"text":" The next lesson but two."}]]},{"t":"h2","id":"where-a-request-goes-wrong","runs":[{"text":"Where a request goes wrong"}]},{"t":"p","runs":[{"text":"The failure modes worth handling before launch, because each one has bitten somebody:"}]},{"t":"p","runs":[{"text":"A missing feature.","b":true},{"text":" The feature store is slow or down. Decide now: fail the request, or predict with an imputed value and flag it. Both are defensible; neither should be discovered at 3am."}]},{"t":"p","runs":[{"text":"An unseen category.","b":true},{"text":" Module 3’s "},{"text":"handle_unknown=\"ignore\"","code":true},{"text":". Without it, the transform raises and the request 500s."}]},{"t":"p","runs":[{"text":"An out-of-range value.","b":true},{"text":" A negative age, an income of 10^12. Validate inputs at the boundary and reject or clip explicitly rather than letting the model extrapolate silently."}]},{"t":"p","runs":[{"text":"The model file will not load.","b":true},{"text":" Version mismatch between the training environment and the serving one. Pin versions, and check that the artefact loads as part of your deploy, before traffic reaches it."}]},{"t":"h2","id":"shadow-and-canary","runs":[{"text":"Shadow and canary"}]},{"t":"p","runs":[{"text":"Two deployment patterns that cost little and prevent most incidents."}]},{"t":"p","runs":[{"text":"Shadow mode.","b":true},{"text":" Run the new model alongside the old one, log both predictions, serve only the old one. After a week you know how the new model behaves on real traffic (including on the inputs your test set never contained) with no risk at all. This is the single most useful practice in this lesson."}]},{"t":"p","runs":[{"text":"Canary.","b":true},{"text":" Route 1% of traffic to the new model, watch the metrics, increase gradually. Standard software practice, and it applies unchanged."}]},{"t":"p","runs":[{"text":"Both require that you can run two models at once and route between them, which is worth building early because everything else depends on it."}]}],"check":{"question":[{"text":"An online fraud service has a mean latency of 35ms and a p99 of 850ms against a 100ms budget. Where should the team look first?"}],"options":[{"text":[{"text":"The model’s inference time, since the tail suggests some predictions require deeper tree traversal."}],"correct":false,"tempting":[{"text":"Tree traversal time varies little between inputs, so it cannot produce a twenty-fold spread between the mean and the tail."}]},{"text":[{"text":"The mean, since 35ms is already a third of the budget and reducing it lowers the tail proportionally."}],"correct":false,"tempting":[{"text":"Tail latency usually has a different cause from the mean, so improving the mean rarely moves the p99 proportionally."}]},{"text":[{"text":"Feature retrieval, since a database round trip that occasionally misses a cache or waits on a slow query is the usual source of a long tail while the mean stays low."}],"correct":true,"tempting":null},{"text":[{"text":"Network time, since 1% of requests may be coming from distant regions."}],"correct":false,"tempting":[{"text":"Geographic variation does add tail latency, but it is bounded and typically far smaller than the 800ms excess described here."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"the-model-is-not-the-system","title":"The model is about 5% of what you have to build","minutes":8},"next":{"slug":"training-serving-skew","title":"When training and serving quietly disagree","minutes":9}}}