{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"what-training-costs","title":"What a training run actually costs","url":"https://addaly.com/learn/how-llms-work/what-training-costs","position":44,"of":78,"minutes":8,"answer":"Two formulas (6ND for training compute and 2N per token for inference) let you estimate the cost of any model from public numbers and check any claim you read.","keyIdea":"Two formulas (6ND for training compute and 2N per token for inference) let you estimate the cost of any model from public numbers and check any claim you read."},"body":{"format":"markdown","text":"## The arithmetic, and why it is worth doing\n\nTraining costs are quoted as impressive numbers with no working. You can do the working yourself, and the result changes how you read announcements.\n\n**Training compute ≈ 6 × N × D**, where N is parameters and D is training tokens. The 6 breaks down as roughly 2 operations per parameter for the forward pass and 4 for the backward pass.\n\nTake an 8B model on 15 trillion tokens:\n\n```text\n6 × 8e9 × 15e12 = 7.2e23 floating-point operations\n```\n\nNow convert to time. A modern accelerator might deliver a few hundred teraflops of *sustained* bf16 throughput on a real workload: well below its headline peak, since utilisation of 35 to 50 per cent is considered good. At 400 teraflops sustained:\n\n```text\n7.2e23 / 4e14 = 1.8e9 seconds = about 500,000 accelerator-hours\n```\n\nOn 1,000 accelerators, roughly three weeks. At cloud rental rates of a few dollars per accelerator-hour, on the order of one to two million dollars of compute for the final run.\n\n## What that number leaves out\n\nThe final run is a fraction of the true cost. Add:\n\n- **Failed and abandoned runs.** Loss spikes, bad hyperparameters, corrupted data shards. Several restarts is normal.\n- **Ablations.** Small-scale experiments to choose the data mixture and architecture. Often more total compute than the final run.\n- **Data pipeline.** Crawling, extracting, filtering and deduplicating tens of terabytes is a large engineering effort in its own right.\n- **Post-training.** Instruction data, human preference collection, safety evaluation. Human annotation at scale is expensive in a way compute is not.\n- **Salaries.** Frequently the largest line, and never in the compute figure.\n\nSo a headline of \"a few million dollars\" for a frontier training run should be read as the compute for one successful pass, not the cost of producing the model.\n\n## Inference, and where the money actually goes\n\n**Inference compute ≈ 2 × N per token.** For an 8B model, 16 billion operations per generated token.\n\nThat sounds trivial next to 7.2e23. It is, per token. But a successful model serves billions of tokens a day, every day, for years. Across a deployed model's lifetime, total inference compute typically exceeds training compute, often by a wide margin.\n\n> Figure: How much serving it takes to match the training run. Bars (trillions of tokens): Pretraining corpus 15; Output needed to equal that compute 45; One year at 10 billion tokens a day 3.65; One year at 120 billion tokens a day 43.8. Training costs about six operations per parameter per token and generating costs about two, so the serving compute overtakes the training compute after three times as many tokens as the model was trained on: 45 trillion here. A product doing ten billion tokens a day takes twelve years to get there; one doing a hundred and twenty billion a day takes one.\n\nThis is the fact behind several industry behaviours that otherwise look strange: the willingness to spend enormous effort distilling models smaller, the investment in serving optimisations from module three, and the preference for sparse architectures. All of them attack the recurring cost rather than the one-off.\n\n## Energy and the numbers people quote\n\nEnergy figures for AI circulate widely and vary by orders of magnitude, which should make you cautious about all of them.\n\nWhat can be said with reasonable confidence: a training run of the scale above consumes energy comparable to a few hundred households for a year, and the per-query energy of a small text request is small: Google published a median figure of about 0.24 watt-hours per text prompt for its assistant in 2025, roughly nine seconds of a television. Multiply by billions of queries and the aggregate is substantial; the per-query figure is not the scary part.\n\nTwo honest caveats. Published figures come from the companies with an interest in them and use methodologies that differ on what to include. And reasoning models that generate thousands of internal tokens per answer cost far more per query than the simple case, so a median figure from one product tells you little about another.\n\n## Using this\n\nWhen you read that a model cost a certain amount, check whether the number is consistent with 6ND and public parameter counts. When a vendor quotes a price per million tokens, compare it against 2N and current hardware costs to see whether it is plausibly above cost or subsidised. And when someone quotes an energy figure, ask what it includes.\n\nNone of this requires inside information. It requires two formulas and public numbers, and it is the difference between reading these announcements and evaluating them."},"check":{"question":"Using 6ND, a 70-billion-parameter model trained on 2 trillion tokens needs about 8.4 × 10²³ operations. A vendor claims the whole model cost $500,000 to produce. What is the sound response?","options":[{"text":"The figure is impossible, since 6ND under-counts the true operation count.","correct":false,"tempting":"The 6ND estimate does omit real overheads, so treating it as a floor is a sensible instinct that leads to the wrong conclusion here."},{"text":"The compute for one successful run may be near that figure, but the claim omits failed runs, ablations, data pipeline work, post-training and salaries.","correct":true},{"text":"The figure is plausible only if the model was trained in fp8 rather than bf16.","correct":false,"tempting":"Lower-precision training genuinely does cut cost, which makes a precision-based reconciliation sound technically alert."},{"text":"The figure must include inference, which dominates the lifetime cost.","correct":false,"tempting":"Inference does dominate over a model's life, so raising it shows real understanding, applied to a claim that was about production cost."}]},"prev":{"slug":"scaling-laws","title":"Scaling laws, and the correction that changed everything","url":"https://addaly.com/learn/how-llms-work/scaling-laws"},"next":{"slug":"fine-tuning-in-practice","title":"Fine-tuning: what it changes and what it cannot","url":"https://addaly.com/learn/how-llms-work/fine-tuning-in-practice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a training run actually costs', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/what-training-costs (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a training run actually costs","courseTitle":"How a Language Model Actually Works","module":{"index":5,"slug":"where-the-weights-come-from","title":"Where the weights come from"},"answer":null,"keyIdea":"Two formulas (6ND for training compute and 2N per token for inference) let you estimate the cost of any model from public numbers and check any claim you read.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"trillions of tokens","title":"How much serving it takes to match the training run","series":[{"label":"Pretraining corpus","value":15},{"label":"Output needed to equal that compute","value":45,"highlight":true},{"label":"One year at 10 billion tokens a day","value":3.65},{"label":"One year at 120 billion tokens a day","value":43.8}],"caption":"Training costs about six operations per parameter per token and generating costs about two, so the serving compute overtakes the training compute after three times as many tokens as the model was trained on: 45 trillion here. A product doing ten billion tokens a day takes twelve years to get there; one doing a hundred and twenty billion a day takes one."},"alt":"Figure: How much serving it takes to match the training run. Bars (trillions of tokens): Pretraining corpus 15; Output needed to equal that compute 45; One year at 10 billion tokens a day 3.65; One year at 120 billion tokens a day 43.8. Training costs about six operations per parameter per token and generating costs about two, so the serving compute overtakes the training compute after three times as many tokens as the model was trained on: 45 trillion here. A product doing ten billion tokens a day takes twelve years to get there; one doing a hundred and twenty billion a day takes one."}},"blocks":[{"t":"h2","id":"the-arithmetic-and-why-it-is-worth-doing","runs":[{"text":"The arithmetic, and why it is worth doing"}]},{"t":"p","runs":[{"text":"Training costs are quoted as impressive numbers with no working. You can do the working yourself, and the result changes how you read announcements."}]},{"t":"p","runs":[{"text":"Training compute ≈ 6 × N × D","b":true},{"text":", where N is parameters and D is training tokens. The 6 breaks down as roughly 2 operations per parameter for the forward pass and 4 for the backward pass."}]},{"t":"p","runs":[{"text":"Take an 8B model on 15 trillion tokens:"}]},{"t":"code","lang":"text","label":"Text","text":"6 × 8e9 × 15e12 = 7.2e23 floating-point operations"},{"t":"p","runs":[{"text":"Now convert to time. A modern accelerator might deliver a few hundred teraflops of "},{"text":"sustained","i":true},{"text":" bf16 throughput on a real workload: well below its headline peak, since utilisation of 35 to 50 per cent is considered good. At 400 teraflops sustained:"}]},{"t":"code","lang":"text","label":"Text","text":"7.2e23 / 4e14 = 1.8e9 seconds = about 500,000 accelerator-hours"},{"t":"p","runs":[{"text":"On 1,000 accelerators, roughly three weeks. At cloud rental rates of a few dollars per accelerator-hour, on the order of one to two million dollars of compute for the final run."}]},{"t":"h2","id":"what-that-number-leaves-out","runs":[{"text":"What that number leaves out"}]},{"t":"p","runs":[{"text":"The final run is a fraction of the true cost. Add:"}]},{"t":"ul","items":[[{"text":"Failed and abandoned runs.","b":true},{"text":" Loss spikes, bad hyperparameters, corrupted data shards. Several restarts is normal."}],[{"text":"Ablations.","b":true},{"text":" Small-scale experiments to choose the data mixture and architecture. Often more total compute than the final run."}],[{"text":"Data pipeline.","b":true},{"text":" Crawling, extracting, filtering and deduplicating tens of terabytes is a large engineering effort in its own right."}],[{"text":"Post-training.","b":true},{"text":" Instruction data, human preference collection, safety evaluation. Human annotation at scale is expensive in a way compute is not."}],[{"text":"Salaries.","b":true},{"text":" Frequently the largest line, and never in the compute figure."}]]},{"t":"p","runs":[{"text":"So a headline of “a few million dollars” for a frontier training run should be read as the compute for one successful pass, not the cost of producing the model."}]},{"t":"h2","id":"inference-and-where-the-money-actually-goes","runs":[{"text":"Inference, and where the money actually goes"}]},{"t":"p","runs":[{"text":"Inference compute ≈ 2 × N per token.","b":true},{"text":" For an 8B model, 16 billion operations per generated token."}]},{"t":"p","runs":[{"text":"That sounds trivial next to 7.2e23. It is, per token. But a successful model serves billions of tokens a day, every day, for years. Across a deployed model’s lifetime, total inference compute typically exceeds training compute, often by a wide margin."}]},{"t":"p","runs":[{"text":"This is the fact behind several industry behaviours that otherwise look strange: the willingness to spend enormous effort distilling models smaller, the investment in serving optimisations from module three, and the preference for sparse architectures. All of them attack the recurring cost rather than the one-off."}]},{"t":"h2","id":"energy-and-the-numbers-people-quote","runs":[{"text":"Energy and the numbers people quote"}]},{"t":"p","runs":[{"text":"Energy figures for AI circulate widely and vary by orders of magnitude, which should make you cautious about all of them."}]},{"t":"p","runs":[{"text":"What can be said with reasonable confidence: a training run of the scale above consumes energy comparable to a few hundred households for a year, and the per-query energy of a small text request is small: Google published a median figure of about 0.24 watt-hours per text prompt for its assistant in 2025, roughly nine seconds of a television. Multiply by billions of queries and the aggregate is substantial; the per-query figure is not the scary part."}]},{"t":"p","runs":[{"text":"Two honest caveats. Published figures come from the companies with an interest in them and use methodologies that differ on what to include. And reasoning models that generate thousands of internal tokens per answer cost far more per query than the simple case, so a median figure from one product tells you little about another."}]},{"t":"h2","id":"using-this","runs":[{"text":"Using this"}]},{"t":"p","runs":[{"text":"When you read that a model cost a certain amount, check whether the number is consistent with 6ND and public parameter counts. When a vendor quotes a price per million tokens, compare it against 2N and current hardware costs to see whether it is plausibly above cost or subsidised. And when someone quotes an energy figure, ask what it includes."}]},{"t":"p","runs":[{"text":"None of this requires inside information. It requires two formulas and public numbers, and it is the difference between reading these announcements and evaluating them."}]}],"check":{"question":[{"text":"Using 6ND, a 70-billion-parameter model trained on 2 trillion tokens needs about 8.4 × 10²³ operations. A vendor claims the whole model cost $500,000 to produce. What is the sound response?"}],"options":[{"text":[{"text":"The figure is impossible, since 6ND under-counts the true operation count."}],"correct":false,"tempting":[{"text":"The 6ND estimate does omit real overheads, so treating it as a floor is a sensible instinct that leads to the wrong conclusion here."}]},{"text":[{"text":"The compute for one successful run may be near that figure, but the claim omits failed runs, ablations, data pipeline work, post-training and salaries."}],"correct":true,"tempting":null},{"text":[{"text":"The figure is plausible only if the model was trained in fp8 rather than bf16."}],"correct":false,"tempting":[{"text":"Lower-precision training genuinely does cut cost, which makes a precision-based reconciliation sound technically alert."}]},{"text":[{"text":"The figure must include inference, which dominates the lifetime cost."}],"correct":false,"tempting":[{"text":"Inference does dominate over a model’s life, so raising it shows real understanding, applied to a claim that was about production cost."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"scaling-laws","title":"Scaling laws, and the correction that changed everything","minutes":9},"next":{"slug":"fine-tuning-in-practice","title":"Fine-tuning: what it changes and what it cannot","minutes":9}}}