Finding your ceiling before your users do

Guessing is the norm and it is always wrong

Ask most people what their app can handle and you get a shrug or a number from a blog post. Both are worthless because the answer depends on your queries, your data volume and your pool sizes.

It takes about an hour to find out for real.

The tools are free

• k6: write the scenario in JavaScript, run one binary. The best default.
• oha and hey: one command, immediate answer, good for a single endpoint.
• wrk: very high throughput from one machine.
• Locust: Python, good for complex user journeys.

oha -z 60s -c 50 https://staging.example.com/search?q=chair

Fifty concurrent connections for sixty seconds. Note it points at staging. Load-testing production is occasionally right, but do it deliberately, at a quiet hour, with a plan for stopping.

Run the same test at increasing concurrency (10, 25, 50, 100, 200) and record throughput and p99 latency at each step. That table is your capacity curve.

The knee

Throughput rises with concurrency, then flattens. Past that point, adding load does not add throughput; it only adds latency, because the extra requests are queueing.

That flattening is the knee, and it is your capacity. Beyond it the system is not “a bit slower”: latency climbs sharply, because queue wait grows without bound while service rate stays fixed. This is why a site under overload does not degrade gracefully by default; it goes from fine to unusable across a narrow band of traffic.

Little’s Law, which is the only formula here

Concurrency = throughput × latency.

At 50 requests per second with 200ms average latency, you have 10 requests in flight at any moment. If your database connection pool has 5 connections, half of them are already waiting for a connection rather than doing work.

Run this backwards to size things. Want to serve 200 requests per second at 100ms? That is 20 in flight, so you need at least 20 pool slots and enough workers to hold 20 concurrent requests. One multiplication, and it replaces most capacity guesswork.

What actually breaks first

In rough order of frequency:

1. Database connections. Almost always. Details in the next lesson.
2. A single unindexed query that was fine at 5,000 rows and is a three-second sequential scan at 5,000,000.
3. Memory, then swapping, then the OOM killer removing your process without explanation.
4. File descriptors, on the default limit of 1024 on some systems.
5. CPU. Genuinely last, and much later than people expect.

Most side projects fall over at 20 requests a second, and it is almost never the language or the framework. It is one query.

Shed load, do not queue it forever

When you are past capacity, you have two choices: queue everything, or refuse some of it.

Queueing feels fairer and is worse. A user who waited 30 seconds has already closed the tab, but your server is still working on their request, still holding a connection, still making the queue longer. You end up doing 100% of the work and delivering 0% of the value.

So: set a request timeout, cap the queue depth, and return 503 with Retry-After when it is exceeded. Serving 90% of people quickly and telling 10% to come back is a better outcome than serving nobody. This is called load shedding and it is the single most useful overload behaviour to configure in advance, because you will not be configuring anything thoughtfully at the time.

Vertical before horizontal

Doubling the size of one machine is a two-minute change with no architectural consequence. Going from one machine to three means sessions must not live in memory, caches must be shared or accept divergence, scheduled jobs must not run three times, and your logs are now in three places.

Scale up until it is expensive, then scale out. A single modern VM with 4 vCPU and 8 GB serves a surprising amount of traffic (thousands of simple requests per second) and almost every product that “needed” to scale out at ten requests a second needed an index.

Do the test again after you change anything

A capacity number ages. New features add queries; new data changes plans. Re-run the same test after any significant release and keep the numbers in a file. Two numbers a month apart tell you more than one number ever will.