Estimating anything to within a factor of three

The skill under every lesson in this module

Enrico Fermi is said to have estimated the yield of the first atomic test by dropping scraps of paper as the blast wave passed and pacing out how far they travelled. The method is general: break a question you cannot answer into factors you can guess to within a factor of three, multiply them, and keep track of the powers of ten. The answer is rarely exact and almost always right about which order of magnitude you are in, and that is the thing you need to know before committing a week or a budget.

Worked: how many tokens is English Wikipedia?

articles              ≈ 7 million
words per article     ≈ 650 (most are short; a few are enormous)
tokens per word       ≈ 1.3
7 × 10^6 × 650 × 1.3  ≈ 6 × 10^9 tokens

The published figures are around four to five billion. Within a factor of 1.5, from three guesses. And the answer is immediately useful: a trillion-token training set is two hundred Wikipedias, which tells you why training corpora are mostly not Wikipedia.

Worked: embedding ten million documents

documents                 10^7
tokens per document       ≈ 500
total tokens              5 × 10^9
embedding model           ≈ 10^8 parameters → 2 × 10^8 FLOPs per token
total FLOPs               10^18
laptop, 10^11 FLOP/s      10^7 s ≈ 4 months
one GPU, 10^14 effective  10^4 s ≈ 3 hours

The plan changes at the fifth line. On a laptop this is not a job; it is a season. Rent a GPU for an afternoon, or use a smaller model, or embed a sample first. Any of these is a good decision, and all of them were invisible before the multiplication.

Worked: what a labelling project costs

items                 10,000
minutes per item      2
hours                 333
rate                  $20 per hour
cost                  ≈ $6,700
plus a second labeller on 20% for agreement:   ≈ $8,000

That is a number a manager can act on, produced in thirty seconds. The course evaluating-ai says why the second labeller is not optional; the arithmetic says what it costs.

Worked: the monthly bill for a retrieval assistant

users                     1,000
queries per user per day  20
tokens per query          ≈ 3,000 (retrieved passages dominate)
tokens per day            6 × 10^7
price                     ≈ $3 per million input tokens
daily cost                ≈ $180  →  monthly ≈ $5,400

If the true answer is $2,000 or $15,000, you were within a factor of three and the estimate did its job, which was to tell you this is thousands a month and not tens or hundreds of thousands. It also shows where the lever is: the 3,000 tokens per query. Halve the retrieved context and halve the bill; nothing else on the list is as easy to move.

Why factors of three are enough

Each guess is off by some factor, and the factors multiply. If every one of k guesses is uncertain by a factor of three, the product is not uncertain by 3^k, because errors in different directions partly cancel. Working in logs, each error is a step of size ln 3 in a random direction, and k random steps travel about √k steps in total. So the product is uncertain by about 3^√k:

k = 4 factors:  3^2   ≈ 9
k = 6 factors:  3^2.4 ≈ 15

Six careless guesses give an answer within an order of magnitude. That is often enough to decide, and when it is not, the estimate tells you which factor to go and measure, since the one you were least sure of dominates the error.

Habits that make estimates good

• Write the factors down. An estimate in your head hides the guess that was wrong; on paper it is the line someone can correct.
• Anchor on numbers you know. A page is about 500 words. A token is about four characters. A GPU-hour is a few dollars. A person labels a few hundred simple items an hour. A laptop does 10^11 FLOP/s and a data-centre GPU a thousand times that. Ten such anchors cover most questions in this field.
• Sanity-check the answer against something independent. If the estimate says a job takes four months, ask whether anyone has done that job in an afternoon. If they have, one of your factors is wrong by a thousand, and it is worth finding which.
• Estimate before, then measure after. A pipeline that runs in a tenth of the estimated time is as interesting as one that takes ten times longer; both mean you misunderstood something, and the misunderstanding is usually the more valuable lesson.

Where it fails

Fermi estimation multiplies independent factors. It cannot see a factor you forgot entirely, and in this field the forgotten factor is usually a limit rather than a rate: the memory that does not fit, the rate limit on an API, the bandwidth ceiling from three lessons ago, the disagreement between two labellers. The estimate tells you the size of the job if nothing is in the way. Then look for what is in the way.