A prediction endpoint is a web service with a large secret

The same shape, with different constants

A model behind an HTTP endpoint is still a process holding a port open. Everything from the first module applies: a proxy in front, a health check, a supervisor, logs with request IDs. What changes is the size of four constants, and every operational surprise in this module comes from one of them.

• Memory. A web process is 100 to 300 MB. A model process is the size of its weights plus the runtime. A gradient-boosted tree is a few megabytes. A small transformer at 8-bit is around 8 GB. An embedding model is a few hundred megabytes.
• Start-up time. Loading weights from disk takes seconds to minutes. A web app starts in one second.
• Concurrency. A web worker spends most of its life waiting on the database, so twenty threads on four cores is fine. A model spends its life computing, so twenty concurrent requests on four cores makes all twenty slow.
• Cost per request. Measured in fractions of a cent for a database query, and in cents for a large model call.

Load once, at start-up, and never inside the handler

The first version of every prediction endpoint contains this bug:

@app.post("/predict")
def predict(req: Item):
    model = joblib.load("model.pkl")   # 400 ms, every single request
    return {"score": float(model.predict_proba([req.features])[0][1])}

Load at import or in a start-up hook, keep it in module scope, and let every request share it. The corollary is that your process now has a warm-up: readiness must stay false until the weights are loaded, or the load balancer will send traffic to a process that is still reading a file.

The same fact explains why scaling to zero hurts more here. A web app’s cold start is a second; a model’s is however long the weights take to load, and the unlucky user waits for all of it.

Workers, threads and the number that is not what you expect

For CPU inference, the arithmetic library is already using every core. Running four worker processes that each start four maths threads gives sixteen threads fighting over four cores, and the result is slower than one worker. This is the most common self-inflicted performance problem in model serving.

OMP_NUM_THREADS=1 gunicorn -w 4 app:app       # 4 workers, 1 thread each
# or
OMP_NUM_THREADS=4 gunicorn -w 1 app:app       # 1 worker, 4 threads

Measure both. Which wins depends on whether your model parallelises well internally, and it is a ten-minute experiment with a load generator.

On a GPU the rule is stricter: one process owns the device, because two processes each allocate their own copy of the weights and then fight for memory. Concurrency comes from batching inside that process, not from more processes.

Do not put a slow model inside a fast request

If inference takes more than a few hundred milliseconds, treat it as background work and apply the queue lesson: accept the request, return a job ID, do the work in a worker, notify when done. Document upload, transcription, batch scoring and image generation all belong here.

What stays inline is anything the user is watching: a search ranking, a classification that gates a form submission, a chat response you stream token by token. For those, streaming is the honest trick: a first token in 300 ms with the rest arriving over four seconds feels dramatically better than four seconds of nothing, even though total time is identical.

The free paths, and they are unusually good here

You do not need a GPU to learn any of this.

• Classical models: scikit-learn, XGBoost or LightGBM exported to ONNX and served with ONNX Runtime. A tree model scores in under a millisecond on a laptop CPU.
• Embeddings: sentence-transformers models are small enough to run on a free-tier box and are the workhorse of search and recommendation.
• Language models locally: llama.cpp and Ollama run quantised models on ordinary hardware. A 7 to 8 billion parameter model at 4-bit needs roughly 5 GB of RAM and produces a handful of tokens per second on a modern CPU: slow, free, and enough to learn every lesson in this module.
• Serving frameworks: FastAPI for a plain endpoint, vLLM or TGI when you have a GPU and need throughput, BentoML or Ray Serve when you need packaging.

Hosted APIs from any provider are the other legitimate answer, and for most small products the right one: the next two lessons are about deciding which, with numbers.

The health check that means something

GET /health returning 200 from a process whose weights failed to load is a lie of exactly the kind the health-check lesson warned about. Make readiness assert that the model object exists and can score one fixed example. Keep that example in the repository and check the output has not changed: it is a smoke test for a corrupted download, a silently swapped file, or a library upgrade that changed a default.