Heavy tails, and the average that describes nobody

The average user does not exist

Take a hundred users of a service. Ninety-five make 2 requests a day. Five make 400.

mean = (95 * 2 + 5 * 400) / 100 = (190 + 2000)/100 = 21.9
median = 2

The average user makes 21.9 requests a day. No user makes 21.9 requests a day. Ninety-five per cent make 2 and five per cent make 400, and the mean sits in an empty region between them, describing nobody.

This is not a contrived example. It is the ordinary shape of usage data, spend per customer, file sizes, session lengths, tokens per request and requests per API key. When somebody says "average users do X", the first question is whether the distribution has a tail, because if it does, the sentence is about a fiction.

What a heavy tail means

A distribution has a heavy tail when extreme values are far more likely than a normal distribution would allow. Under a normal, a value five standard deviations out happens roughly once in 3.5 million. Under a heavy-tailed distribution, it happens often enough to matter every week.

The sharpest version is the power law, where the probability of a value above x falls as x^(−α). City sizes, word frequencies, file sizes, wealth and network degrees all approximately follow one. When α ≤ 2 the variance is infinite — not large, infinite — and when α ≤ 1 the mean is infinite too. Computing a sample mean from such data gives you a number, and that number does not converge as you collect more data. It keeps drifting upward as bigger examples arrive.

That is worth sitting with. Some quantities have no meaningful average, and the sample average of them is not an estimate of anything.

The latency case, which you will meet

Response times are always right-skewed. A service with a 40 ms mean routinely has a 400 ms 99th percentile, because a small fraction of requests hit a cold cache, a garbage collection pause, or a slow dependency.

The consequence people miss: a page that makes 20 backend calls will show its user the slowest of the 20. If each call independently exceeds 400 ms one per cent of the time,

P(all 20 fast) = 0.99^20 = 0.818
P(at least one slow) = 18.2%

So a "1 per cent tail" at the service level is an 18 per cent tail at the page level. This is why serious latency work targets the 99th and 99.9th percentiles rather than the mean, and why an average latency in a dashboard is close to useless as a user-experience measure.

Cost, where the same arithmetic bites

Token usage per request is heavy-tailed. Most requests are short; a few paste in an entire document. Budgeting on the mean request size understates cost, because the mean is itself unstable — one very large request moves it, and next month's mean will differ.

The defensible approach is to budget from percentiles and from the total, not from the average: measure total tokens per day directly, and separately cap the per-request maximum. A hard input cap converts an unbounded tail into a bounded one, which is the only thing that makes the cost predictable at all.

How to tell if you have one

Three checks, all cheap:

1. Compare mean and median. If the mean is far above the median, the tail is on the right. A ratio above about 1.5 is worth investigating.
2. Plot the histogram of the logs. If the log-histogram looks roughly symmetric, you have something log-normal-ish and should work in log space.
3. Plot the survival function on log-log axes. Plot the fraction of values above x against x, both on log scales. A power law appears as a straight line. This is the standard diagnostic, and it takes four lines.

import numpy as np
x = np.sort(data)[::-1]
frac = np.arange(1, len(x)+1) / len(x)
# plot log(x) against log(frac); a straight line indicates a power law

Be careful with the third: many distributions look straight-ish over one or two decades, and claims of power laws in the literature have a long history of being overturned by more careful fitting. Report it as "consistent with" rather than "is".

What to do instead of the mean

• Report percentiles. Median, 90th, 99th. Three numbers describe a skewed distribution far better than a mean and a standard deviation, which describe it wrongly.
• Work in log space. For log-normal-ish quantities, take logs, do the arithmetic, and transform back. The geometric mean is often the right central summary.
• Trim or winsorise, and say so. Dropping the top 1 per cent stabilises an estimate at the cost of ignoring exactly the cases that may matter most. It is a legitimate choice that must be disclosed, because the trimmed mean of a heavy-tailed quantity is a different quantity.
• Separate the populations. Sometimes the tail is not a tail but a second group — bots, batch jobs, one enterprise customer. Splitting them gives two distributions that each behave, and is more informative than any robust statistic applied to the mixture.

The rule to keep

Before quoting a mean, compare it with the median. If they differ substantially, the mean is being carried by a few extreme values and every plan built on it — capacity, cost, expected quality — is built on a number that describes none of your cases.