{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"a-model-is-a-process-with-weights","title":"A prediction endpoint is a web service with a large secret","url":"https://addaly.com/learn/cloud-and-shipping/a-model-is-a-process-with-weights","position":49,"of":74,"minutes":9,"answer":"A model endpoint is an ordinary web service whose memory, start-up time, concurrency and per-request cost are one to three orders of magnitude larger, so load weights once.","keyIdea":"A model endpoint is an ordinary web service whose memory, start-up time, concurrency and per-request cost are one to three orders of magnitude larger, so load weights once, size workers against cores rather than against waiting, and push anything slow into a queue."},"body":{"format":"markdown","text":"## The same shape, with different constants\n\nA model behind an HTTP endpoint is still a process holding a port open. Everything from the first module applies: a proxy in front, a health check, a supervisor, logs with request IDs. What changes is the size of four constants, and every operational surprise in this module comes from one of them.\n\n- **Memory.** A web process is 100 to 300 MB. A model process is the size of its weights plus the runtime. A gradient-boosted tree is a few megabytes. A small transformer at 8-bit is around 8 GB. An embedding model is a few hundred megabytes.\n- **Start-up time.** Loading weights from disk takes seconds to minutes. A web app starts in one second.\n- **Concurrency.** A web worker spends most of its life waiting on the database, so twenty threads on four cores is fine. A model spends its life computing, so twenty concurrent requests on four cores makes all twenty slow.\n- **Cost per request.** Measured in fractions of a cent for a database query, and in cents for a large model call.\n\n## Load once, at start-up, and never inside the handler\n\nThe first version of every prediction endpoint contains this bug:\n\n```python\n@app.post(\"/predict\")\ndef predict(req: Item):\n    model = joblib.load(\"model.pkl\")   # 400 ms, every single request\n    return {\"score\": float(model.predict_proba([req.features])[0][1])}\n```\n\nLoad at import or in a start-up hook, keep it in module scope, and let every request share it. The corollary is that your process now has a warm-up: readiness must stay false until the weights are loaded, or the load balancer will send traffic to a process that is still reading a file.\n\nThe same fact explains why scaling to zero hurts more here. A web app's cold start is a second; a model's is however long the weights take to load, and the unlucky user waits for all of it.\n\n## Workers, threads and the number that is not what you expect\n\nFor CPU inference, the arithmetic library is already using every core. Running four worker processes that each start four maths threads gives sixteen threads fighting over four cores, and the result is slower than one worker. This is the most common self-inflicted performance problem in model serving.\n\n```bash\nOMP_NUM_THREADS=1 gunicorn -w 4 app:app       # 4 workers, 1 thread each\n# or\nOMP_NUM_THREADS=4 gunicorn -w 1 app:app       # 1 worker, 4 threads\n```\n\nMeasure both. Which wins depends on whether your model parallelises well internally, and it is a ten-minute experiment with a load generator.\n\nOn a GPU the rule is stricter: one process owns the device, because two processes each allocate their own copy of the weights and then fight for memory. Concurrency comes from batching inside that process, not from more processes.\n\n## Do not put a slow model inside a fast request\n\nIf inference takes more than a few hundred milliseconds, treat it as background work and apply the queue lesson: accept the request, return a job ID, do the work in a worker, notify when done. Document upload, transcription, batch scoring and image generation all belong here.\n\nWhat stays inline is anything the user is watching: a search ranking, a classification that gates a form submission, a chat response you stream token by token. For those, streaming is the honest trick: a first token in 300 ms with the rest arriving over four seconds feels dramatically better than four seconds of nothing, even though total time is identical.\n\n## The free paths, and they are unusually good here\n\nYou do not need a GPU to learn any of this.\n\n- **Classical models:** scikit-learn, XGBoost or LightGBM exported to ONNX and served with ONNX Runtime. A tree model scores in under a millisecond on a laptop CPU.\n- **Embeddings:** `sentence-transformers` models are small enough to run on a free-tier box and are the workhorse of search and recommendation.\n- **Language models locally:** llama.cpp and Ollama run quantised models on ordinary hardware. A 7 to 8 billion parameter model at 4-bit needs roughly 5 GB of RAM and produces a handful of tokens per second on a modern CPU: slow, free, and enough to learn every lesson in this module.\n- **Serving frameworks:** FastAPI for a plain endpoint, vLLM or TGI when you have a GPU and need throughput, BentoML or Ray Serve when you need packaging.\n\nHosted APIs from any provider are the other legitimate answer, and for most small products the right one: the next two lessons are about deciding which, with numbers.\n\n## The health check that means something\n\n`GET /health` returning 200 from a process whose weights failed to load is a lie of exactly the kind the health-check lesson warned about. Make readiness assert that the model object exists and can score one fixed example. Keep that example in the repository and check the output has not changed: it is a smoke test for a corrupted download, a silently swapped file, or a library upgrade that changed a default."},"check":{"question":"A CPU-served model runs at 40 ms per request with one worker. The team moves to four workers on a four-core machine and latency rises to 130 ms under the same load. What is the most likely explanation?","options":[{"text":"Each worker starts its own arithmetic threads, so sixteen threads contend for four cores","correct":true},{"text":"The four workers each hold a copy of the weights, exhausting memory and causing swapping","correct":false,"tempting":"Memory duplication is real and worth watching, but swapping produces far worse and more erratic latency than a threefold rise, and would show in the machine's swap counters."},{"text":"The load balancer is now spreading requests, so each worker's cache of recent inputs is less effective","correct":false,"tempting":"Inference does not usually benefit from a per-worker input cache, and cache locality could not account for a systematic slowdown of this size."},{"text":"Four workers exceed the model's supported batch size, forcing requests to queue","correct":false,"tempting":"Confuses process concurrency with batching; separate processes do not share a batch, and no batch limit is being exceeded."}]},"prev":{"slug":"handing-it-over","title":"Writing it down for whoever is next","url":"https://addaly.com/learn/cloud-and-shipping/handing-it-over"},"next":{"slug":"where-inference-time-goes","title":"Where inference time actually goes","url":"https://addaly.com/learn/cloud-and-shipping/where-inference-time-goes"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A prediction endpoint is a web service with a large secret', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/a-model-is-a-process-with-weights (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A prediction endpoint is a web service with a large secret","courseTitle":"Shipping It","module":{"index":6,"slug":"shipping-a-model","title":"Shipping a model, not just a web app"},"answer":null,"keyIdea":"A model endpoint is an ordinary web service whose memory, start-up time, concurrency and per-request cost are one to three orders of magnitude larger, so load weights once, size workers against cores rather than against waiting, and push anything slow into a queue.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-same-shape-with-different-constants","runs":[{"text":"The same shape, with different constants"}]},{"t":"p","runs":[{"text":"A model behind an HTTP endpoint is still a process holding a port open. Everything from the first module applies: a proxy in front, a health check, a supervisor, logs with request IDs. What changes is the size of four constants, and every operational surprise in this module comes from one of them."}]},{"t":"ul","items":[[{"text":"Memory.","b":true},{"text":" A web process is 100 to 300 MB. A model process is the size of its weights plus the runtime. A gradient-boosted tree is a few megabytes. A small transformer at 8-bit is around 8 GB. An embedding model is a few hundred megabytes."}],[{"text":"Start-up time.","b":true},{"text":" Loading weights from disk takes seconds to minutes. A web app starts in one second."}],[{"text":"Concurrency.","b":true},{"text":" A web worker spends most of its life waiting on the database, so twenty threads on four cores is fine. A model spends its life computing, so twenty concurrent requests on four cores makes all twenty slow."}],[{"text":"Cost per request.","b":true},{"text":" Measured in fractions of a cent for a database query, and in cents for a large model call."}]]},{"t":"h2","id":"load-once-at-start-up-and-never-inside-the-handler","runs":[{"text":"Load once, at start-up, and never inside the handler"}]},{"t":"p","runs":[{"text":"The first version of every prediction endpoint contains this bug:"}]},{"t":"code","lang":"python","label":"Python","text":"@app.post(\"/predict\")\ndef predict(req: Item):\n    model = joblib.load(\"model.pkl\")   # 400 ms, every single request\n    return {\"score\": float(model.predict_proba([req.features])[0][1])}"},{"t":"p","runs":[{"text":"Load at import or in a start-up hook, keep it in module scope, and let every request share it. The corollary is that your process now has a warm-up: readiness must stay false until the weights are loaded, or the load balancer will send traffic to a process that is still reading a file."}]},{"t":"p","runs":[{"text":"The same fact explains why scaling to zero hurts more here. A web app’s cold start is a second; a model’s is however long the weights take to load, and the unlucky user waits for all of it."}]},{"t":"h2","id":"workers-threads-and-the-number-that-is-not-what-you-expect","runs":[{"text":"Workers, threads and the number that is not what you expect"}]},{"t":"p","runs":[{"text":"For CPU inference, the arithmetic library is already using every core. Running four worker processes that each start four maths threads gives sixteen threads fighting over four cores, and the result is slower than one worker. This is the most common self-inflicted performance problem in model serving."}]},{"t":"code","lang":"bash","label":"Bash","text":"OMP_NUM_THREADS=1 gunicorn -w 4 app:app       # 4 workers, 1 thread each\n# or\nOMP_NUM_THREADS=4 gunicorn -w 1 app:app       # 1 worker, 4 threads"},{"t":"p","runs":[{"text":"Measure both. Which wins depends on whether your model parallelises well internally, and it is a ten-minute experiment with a load generator."}]},{"t":"p","runs":[{"text":"On a GPU the rule is stricter: one process owns the device, because two processes each allocate their own copy of the weights and then fight for memory. Concurrency comes from batching inside that process, not from more processes."}]},{"t":"h2","id":"do-not-put-a-slow-model-inside-a-fast-request","runs":[{"text":"Do not put a slow model inside a fast request"}]},{"t":"p","runs":[{"text":"If inference takes more than a few hundred milliseconds, treat it as background work and apply the queue lesson: accept the request, return a job ID, do the work in a worker, notify when done. Document upload, transcription, batch scoring and image generation all belong here."}]},{"t":"p","runs":[{"text":"What stays inline is anything the user is watching: a search ranking, a classification that gates a form submission, a chat response you stream token by token. For those, streaming is the honest trick: a first token in 300 ms with the rest arriving over four seconds feels dramatically better than four seconds of nothing, even though total time is identical."}]},{"t":"h2","id":"the-free-paths-and-they-are-unusually-good-here","runs":[{"text":"The free paths, and they are unusually good here"}]},{"t":"p","runs":[{"text":"You do not need a GPU to learn any of this."}]},{"t":"ul","items":[[{"text":"Classical models:","b":true},{"text":" scikit-learn, XGBoost or LightGBM exported to ONNX and served with ONNX Runtime. A tree model scores in under a millisecond on a laptop CPU."}],[{"text":"Embeddings:","b":true},{"text":" "},{"text":"sentence-transformers","code":true},{"text":" models are small enough to run on a free-tier box and are the workhorse of search and recommendation."}],[{"text":"Language models locally:","b":true},{"text":" llama.cpp and Ollama run quantised models on ordinary hardware. A 7 to 8 billion parameter model at 4-bit needs roughly 5 GB of RAM and produces a handful of tokens per second on a modern CPU: slow, free, and enough to learn every lesson in this module."}],[{"text":"Serving frameworks:","b":true},{"text":" FastAPI for a plain endpoint, vLLM or TGI when you have a GPU and need throughput, BentoML or Ray Serve when you need packaging."}]]},{"t":"p","runs":[{"text":"Hosted APIs from any provider are the other legitimate answer, and for most small products the right one: the next two lessons are about deciding which, with numbers."}]},{"t":"h2","id":"the-health-check-that-means-something","runs":[{"text":"The health check that means something"}]},{"t":"p","runs":[{"text":"GET /health","code":true},{"text":" returning 200 from a process whose weights failed to load is a lie of exactly the kind the health-check lesson warned about. Make readiness assert that the model object exists and can score one fixed example. Keep that example in the repository and check the output has not changed: it is a smoke test for a corrupted download, a silently swapped file, or a library upgrade that changed a default."}]}],"check":{"question":[{"text":"A CPU-served model runs at 40 ms per request with one worker. The team moves to four workers on a four-core machine and latency rises to 130 ms under the same load. What is the most likely explanation?"}],"options":[{"text":[{"text":"Each worker starts its own arithmetic threads, so sixteen threads contend for four cores"}],"correct":true,"tempting":null},{"text":[{"text":"The four workers each hold a copy of the weights, exhausting memory and causing swapping"}],"correct":false,"tempting":[{"text":"Memory duplication is real and worth watching, but swapping produces far worse and more erratic latency than a threefold rise, and would show in the machine’s swap counters."}]},{"text":[{"text":"The load balancer is now spreading requests, so each worker’s cache of recent inputs is less effective"}],"correct":false,"tempting":[{"text":"Inference does not usually benefit from a per-worker input cache, and cache locality could not account for a systematic slowdown of this size."}]},{"text":[{"text":"Four workers exceed the model’s supported batch size, forcing requests to queue"}],"correct":false,"tempting":[{"text":"Confuses process concurrency with batching; separate processes do not share a batch, and no batch limit is being exceeded."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"handing-it-over","title":"Writing it down for whoever is next","minutes":9},"next":{"slug":"where-inference-time-goes","title":"Where inference time actually goes","minutes":10}}}