Four numbers that describe any service

Logs answer “what happened to this request”. Metrics answer “how is it going”

They are different tools and people try to make one do the other’s job. Grepping logs to work out whether the site is slower than last week is miserable; a graph answers it in a second. Conversely, a graph will never tell you why request a7f3 failed.

The four signals

Google’s site reliability book names four, and after fifteen years nobody has improved on them:

1. Latency: how long requests take. Split successful from failed, because a fast 500 makes your latency graph look wonderful.
2. Traffic: requests per second, by endpoint.
3. Errors: the rate of failures, as a proportion, not a count. Forty errors means nothing until you know whether it was out of a hundred or a million.
4. Saturation: how full the constrained resource is. Usually not CPU. Usually the database connection pool, the queue depth, or memory.

Two shorthands you will meet: RED (rate, errors, duration) for services handling requests, and USE (utilisation, saturation, errors) for resources like disks and pools. They are the same four ideas from two sides.

Start with three metrics, not thirty. A dashboard with forty panels is one nobody reads.

The average is lying to you

Take one hundred requests. Ninety-nine take 100ms. One takes 5 seconds.

• Mean: 149ms. Looks fine.
• p50 (median): 100ms. Looks fine.
• p99: 5,000ms.

The mean and median both hide the request that made somebody close the tab. Latency is not normally distributed: it has a long tail, and the tail is where the pain is. Always look at p50 alongside p95 or p99. p50 tells you what the typical experience is; p99 tells you what your worst hour looks like.

Now the part that surprises people. Suppose a page makes 20 backend calls, and each has a 1% chance of being slow. The chance that at least one of them is slow is 1 - 0.99^20, which is about 18%. A p99 at the service level is close to a p82 at the page level. This is why tail latency matters far more than its name suggests, and why “only 1% of requests” is not the reassurance it sounds like.

Percentiles cannot be averaged

This one is worth memorising because it is done wrong constantly. You cannot take the p99 from each of your four servers and average them to get the fleet p99. You cannot average five one-minute p99s to get a five-minute p99. The result is a number with no meaning that will happily be plotted.

To aggregate correctly you need the underlying distribution, which is why metrics systems record latency as a histogram (a set of counters, one per bucket) and compute percentiles from the summed buckets at query time. If your monitoring only stores a pre-computed average per minute, you cannot get percentiles out of it later. Choose the histogram at the start.

Counters, gauges, histograms

• Counter: only goes up. Requests served, errors, bytes. You graph its rate, not its value.
• Gauge: goes up and down. Queue depth, memory in use, connections open.
• Histogram: a distribution. Latency, payload size.

Picking the wrong one is a common beginner error: recording latency as a gauge gives you the last value observed, which is noise.

Cardinality, and how a metric becomes a bill

Every distinct combination of label values is a separate stored series. http_requests_total{path, status, method} with 50 paths, 8 statuses and 4 methods is 1,600 series. Fine.

Add user_id and you have 1,600 series per user. At 50,000 users that is 80 million series. Your metrics store falls over, or your vendor’s bill arrives with a number on it you did not expect, because most of them charge per series.

The rule: labels are for values from a small, fixed set. Path template (/users/:id, never /users/8412), status, region, method. Never user IDs, request IDs, email addresses, or full URLs with query strings. If you want per-user detail, that is what logs and traces are for.

The free path

Nothing here requires a vendor.

• Prometheus scrapes a /metrics endpoint your app exposes and stores the series. Grafana draws them. Both free, both self-hostable on the same small VM as your app.
• Grafana Cloud has a free tier that is generous for a small project, if you would rather not run the storage.
• VictoriaMetrics is a lighter drop-in replacement for Prometheus storage.

Datadog and New Relic are excellent and priced per host plus per custom metric, which is how a monitoring bill quietly becomes larger than the hosting bill. Know what you are agreeing to before you add a label.