{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"metrics-that-mean-something","title":"Four numbers that describe any service","url":"https://addaly.com/learn/cloud-and-shipping/metrics-that-mean-something","position":22,"of":74,"minutes":9,"answer":"Averages hide exactly the requests you care about, so measure latency as percentiles from histograms and keep the number of distinct label values small.","keyIdea":"Averages hide exactly the requests you care about, so measure latency as percentiles from histograms and keep the number of distinct label values small."},"body":{"format":"markdown","text":"## Logs answer \"what happened to this request\". Metrics answer \"how is it going\"\n\nThey are different tools and people try to make one do the other's job. Grepping logs to work out whether the site is slower than last week is miserable; a graph answers it in a second. Conversely, a graph will never tell you why request `a7f3` failed.\n\n## The four signals\n\nGoogle's site reliability book names four, and after fifteen years nobody has improved on them:\n\n1. **Latency**: how long requests take. Split successful from failed, because a fast 500 makes your latency graph look wonderful.\n2. **Traffic**: requests per second, by endpoint.\n3. **Errors**: the rate of failures, as a proportion, not a count. Forty errors means nothing until you know whether it was out of a hundred or a million.\n4. **Saturation**: how full the constrained resource is. Usually not CPU. Usually the database connection pool, the queue depth, or memory.\n\nTwo shorthands you will meet: **RED** (rate, errors, duration) for services handling requests, and **USE** (utilisation, saturation, errors) for resources like disks and pools. They are the same four ideas from two sides.\n\nStart with three metrics, not thirty. A dashboard with forty panels is one nobody reads.\n\n## The average is lying to you\n\nTake one hundred requests. Ninety-nine take 100ms. One takes 5 seconds.\n\n- Mean: 149ms. Looks fine.\n- p50 (median): 100ms. Looks fine.\n- p99: 5,000ms.\n\nThe mean and median both hide the request that made somebody close the tab. Latency is not normally distributed: it has a long tail, and the tail is where the pain is. Always look at p50 alongside p95 or p99. p50 tells you what the typical experience is; p99 tells you what your worst hour looks like.\n\n> Figure: A hundred requests: ninety-nine at 100 ms, one at 5 seconds. Bars (ms): Mean 149; p50 (median) 100; p99 5000. The mean and the median both look fine. The request that made somebody close the tab only appears at p99, which is why latency is recorded as a histogram and read as percentiles.\n\nNow the part that surprises people. Suppose a page makes 20 backend calls, and each has a 1% chance of being slow. The chance that *at least one* of them is slow is `1 - 0.99^20`, which is about **18%**. A p99 at the service level is close to a p82 at the page level. This is why tail latency matters far more than its name suggests, and why \"only 1% of requests\" is not the reassurance it sounds like.\n\n## Percentiles cannot be averaged\n\nThis one is worth memorising because it is done wrong constantly. You cannot take the p99 from each of your four servers and average them to get the fleet p99. You cannot average five one-minute p99s to get a five-minute p99. The result is a number with no meaning that will happily be plotted.\n\nTo aggregate correctly you need the underlying distribution, which is why metrics systems record latency as a **histogram** (a set of counters, one per bucket) and compute percentiles from the summed buckets at query time. If your monitoring only stores a pre-computed average per minute, you cannot get percentiles out of it later. Choose the histogram at the start.\n\n## Counters, gauges, histograms\n\n- **Counter**: only goes up. Requests served, errors, bytes. You graph its rate, not its value.\n- **Gauge**: goes up and down. Queue depth, memory in use, connections open.\n- **Histogram**: a distribution. Latency, payload size.\n\nPicking the wrong one is a common beginner error: recording latency as a gauge gives you the last value observed, which is noise.\n\n## Cardinality, and how a metric becomes a bill\n\nEvery distinct combination of label values is a separate stored series. `http_requests_total{path, status, method}` with 50 paths, 8 statuses and 4 methods is 1,600 series. Fine.\n\nAdd `user_id` and you have 1,600 series per user. At 50,000 users that is 80 million series. Your metrics store falls over, or your vendor's bill arrives with a number on it you did not expect, because most of them charge per series.\n\nThe rule: **labels are for values from a small, fixed set.** Path template (`/users/:id`, never `/users/8412`), status, region, method. Never user IDs, request IDs, email addresses, or full URLs with query strings. If you want per-user detail, that is what logs and traces are for.\n\n## The free path\n\nNothing here requires a vendor.\n\n- **Prometheus** scrapes a `/metrics` endpoint your app exposes and stores the series. **Grafana** draws them. Both free, both self-hostable on the same small VM as your app.\n- **Grafana Cloud** has a free tier that is generous for a small project, if you would rather not run the storage.\n- **VictoriaMetrics** is a lighter drop-in replacement for Prometheus storage.\n\nDatadog and New Relic are excellent and priced per host plus per custom metric, which is how a monitoring bill quietly becomes larger than the hosting bill. Know what you are agreeing to before you add a label."},"check":{"question":"Your dashboard shows a mean response time of 149ms, steady for weeks. Support keeps receiving complaints that the site \"hangs sometimes\". The mean has not moved. What is the most likely explanation?","options":[{"text":"The complaints are about client-side rendering, which server metrics cannot see","correct":false,"tempting":"Possible in general, and worth checking, but it abandons the server-side data before reading it properly, and the mean is specifically constructed to hide this case."},{"text":"Users are on slow connections, so their experienced latency differs from the server's","correct":false,"tempting":"Real, but it would affect everyone consistently rather than producing intermittent hangs, and it does not explain why the metric looks healthy."},{"text":"The mean is computed over too long a window, so short spikes are smoothed away","correct":false,"tempting":"Close, and shortening the window helps a little, but even at one-minute resolution a mean over a long-tailed distribution still hides a small fraction of very slow requests."},{"text":"A small fraction of requests are very slow, which barely moves the mean: the p99 would show it, and the mean is the wrong statistic for a long-tailed distribution","correct":true}]},"prev":{"slug":"where-logs-go-and-what-they-cost","title":"Where the logs go, and what they cost","url":"https://addaly.com/learn/cloud-and-shipping/where-logs-go-and-what-they-cost"},"next":{"slug":"dashboards-people-actually-read","title":"One screen that answers the question","url":"https://addaly.com/learn/cloud-and-shipping/dashboards-people-actually-read"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Four numbers that describe any service', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/metrics-that-mean-something (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Four numbers that describe any service","courseTitle":"Shipping It","module":{"index":3,"slug":"knowing-what-it-is-doing","title":"Knowing what it is doing"},"answer":null,"keyIdea":"Averages hide exactly the requests you care about, so measure latency as percentiles from histograms and keep the number of distinct label values small.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"ms","title":"A hundred requests: ninety-nine at 100 ms, one at 5 seconds","series":[{"label":"Mean","value":149},{"label":"p50 (median)","value":100},{"label":"p99","value":5000,"highlight":true}],"caption":"The mean and the median both look fine. The request that made somebody close the tab only appears at p99, which is why latency is recorded as a histogram and read as percentiles."},"alt":"Figure: A hundred requests: ninety-nine at 100 ms, one at 5 seconds. Bars (ms): Mean 149; p50 (median) 100; p99 5000. The mean and the median both look fine. The request that made somebody close the tab only appears at p99, which is why latency is recorded as a histogram and read as percentiles."}},"blocks":[{"t":"h2","id":"logs-answer-what-happened-to-this-request-metrics-answer-how","runs":[{"text":"Logs answer “what happened to this request”. Metrics answer “how is it going”"}]},{"t":"p","runs":[{"text":"They are different tools and people try to make one do the other’s job. Grepping logs to work out whether the site is slower than last week is miserable; a graph answers it in a second. Conversely, a graph will never tell you why request "},{"text":"a7f3","code":true},{"text":" failed."}]},{"t":"h2","id":"the-four-signals","runs":[{"text":"The four signals"}]},{"t":"p","runs":[{"text":"Google’s site reliability book names four, and after fifteen years nobody has improved on them:"}]},{"t":"ol","start":1,"items":[[{"text":"Latency","b":true},{"text":": how long requests take. Split successful from failed, because a fast 500 makes your latency graph look wonderful."}],[{"text":"Traffic","b":true},{"text":": requests per second, by endpoint."}],[{"text":"Errors","b":true},{"text":": the rate of failures, as a proportion, not a count. Forty errors means nothing until you know whether it was out of a hundred or a million."}],[{"text":"Saturation","b":true},{"text":": how full the constrained resource is. Usually not CPU. Usually the database connection pool, the queue depth, or memory."}]]},{"t":"p","runs":[{"text":"Two shorthands you will meet: "},{"text":"RED","b":true},{"text":" (rate, errors, duration) for services handling requests, and "},{"text":"USE","b":true},{"text":" (utilisation, saturation, errors) for resources like disks and pools. They are the same four ideas from two sides."}]},{"t":"p","runs":[{"text":"Start with three metrics, not thirty. A dashboard with forty panels is one nobody reads."}]},{"t":"h2","id":"the-average-is-lying-to-you","runs":[{"text":"The average is lying to you"}]},{"t":"p","runs":[{"text":"Take one hundred requests. Ninety-nine take 100ms. One takes 5 seconds."}]},{"t":"ul","items":[[{"text":"Mean: 149ms. Looks fine."}],[{"text":"p50 (median): 100ms. Looks fine."}],[{"text":"p99: 5,000ms."}]]},{"t":"p","runs":[{"text":"The mean and median both hide the request that made somebody close the tab. Latency is not normally distributed: it has a long tail, and the tail is where the pain is. Always look at p50 alongside p95 or p99. p50 tells you what the typical experience is; p99 tells you what your worst hour looks like."}]},{"t":"p","runs":[{"text":"Now the part that surprises people. Suppose a page makes 20 backend calls, and each has a 1% chance of being slow. The chance that "},{"text":"at least one","i":true},{"text":" of them is slow is "},{"text":"1 - 0.99^20","code":true},{"text":", which is about "},{"text":"18%","b":true},{"text":". A p99 at the service level is close to a p82 at the page level. This is why tail latency matters far more than its name suggests, and why “only 1% of requests” is not the reassurance it sounds like."}]},{"t":"h2","id":"percentiles-cannot-be-averaged","runs":[{"text":"Percentiles cannot be averaged"}]},{"t":"p","runs":[{"text":"This one is worth memorising because it is done wrong constantly. You cannot take the p99 from each of your four servers and average them to get the fleet p99. You cannot average five one-minute p99s to get a five-minute p99. The result is a number with no meaning that will happily be plotted."}]},{"t":"p","runs":[{"text":"To aggregate correctly you need the underlying distribution, which is why metrics systems record latency as a "},{"text":"histogram","b":true},{"text":" (a set of counters, one per bucket) and compute percentiles from the summed buckets at query time. If your monitoring only stores a pre-computed average per minute, you cannot get percentiles out of it later. Choose the histogram at the start."}]},{"t":"h2","id":"counters-gauges-histograms","runs":[{"text":"Counters, gauges, histograms"}]},{"t":"ul","items":[[{"text":"Counter","b":true},{"text":": only goes up. Requests served, errors, bytes. You graph its rate, not its value."}],[{"text":"Gauge","b":true},{"text":": goes up and down. Queue depth, memory in use, connections open."}],[{"text":"Histogram","b":true},{"text":": a distribution. Latency, payload size."}]]},{"t":"p","runs":[{"text":"Picking the wrong one is a common beginner error: recording latency as a gauge gives you the last value observed, which is noise."}]},{"t":"h2","id":"cardinality-and-how-a-metric-becomes-a-bill","runs":[{"text":"Cardinality, and how a metric becomes a bill"}]},{"t":"p","runs":[{"text":"Every distinct combination of label values is a separate stored series. "},{"text":"http_requests_total{path, status, method}","code":true},{"text":" with 50 paths, 8 statuses and 4 methods is 1,600 series. Fine."}]},{"t":"p","runs":[{"text":"Add "},{"text":"user_id","code":true},{"text":" and you have 1,600 series per user. At 50,000 users that is 80 million series. Your metrics store falls over, or your vendor’s bill arrives with a number on it you did not expect, because most of them charge per series."}]},{"t":"p","runs":[{"text":"The rule: "},{"text":"labels are for values from a small, fixed set.","b":true},{"text":" Path template ("},{"text":"/users/:id","code":true},{"text":", never "},{"text":"/users/8412","code":true},{"text":"), status, region, method. Never user IDs, request IDs, email addresses, or full URLs with query strings. If you want per-user detail, that is what logs and traces are for."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"Nothing here requires a vendor."}]},{"t":"ul","items":[[{"text":"Prometheus","b":true},{"text":" scrapes a "},{"text":"/metrics","code":true},{"text":" endpoint your app exposes and stores the series. "},{"text":"Grafana","b":true},{"text":" draws them. Both free, both self-hostable on the same small VM as your app."}],[{"text":"Grafana Cloud","b":true},{"text":" has a free tier that is generous for a small project, if you would rather not run the storage."}],[{"text":"VictoriaMetrics","b":true},{"text":" is a lighter drop-in replacement for Prometheus storage."}]]},{"t":"p","runs":[{"text":"Datadog and New Relic are excellent and priced per host plus per custom metric, which is how a monitoring bill quietly becomes larger than the hosting bill. Know what you are agreeing to before you add a label."}]}],"check":{"question":[{"text":"Your dashboard shows a mean response time of 149ms, steady for weeks. Support keeps receiving complaints that the site “hangs sometimes”. The mean has not moved. What is the most likely explanation?"}],"options":[{"text":[{"text":"The complaints are about client-side rendering, which server metrics cannot see"}],"correct":false,"tempting":[{"text":"Possible in general, and worth checking, but it abandons the server-side data before reading it properly, and the mean is specifically constructed to hide this case."}]},{"text":[{"text":"Users are on slow connections, so their experienced latency differs from the server’s"}],"correct":false,"tempting":[{"text":"Real, but it would affect everyone consistently rather than producing intermittent hangs, and it does not explain why the metric looks healthy."}]},{"text":[{"text":"The mean is computed over too long a window, so short spikes are smoothed away"}],"correct":false,"tempting":[{"text":"Close, and shortening the window helps a little, but even at one-minute resolution a mean over a long-tailed distribution still hides a small fraction of very slow requests."}]},{"text":[{"text":"A small fraction of requests are very slow, which barely moves the mean: the p99 would show it, and the mean is the wrong statistic for a long-tailed distribution"}],"correct":true,"tempting":null}],"order":[1,0,2,3],"why":null},"prev":{"slug":"where-logs-go-and-what-they-cost","title":"Where the logs go, and what they cost","minutes":9},"next":{"slug":"dashboards-people-actually-read","title":"One screen that answers the question","minutes":8}}}