{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"caching-and-batching","title":"The two discounts almost nobody uses","url":"https://addaly.com/learn/choosing-your-tools/caching-and-batching","position":33,"of":76,"minutes":8,"answer":"Put every stable block at the front of a request so prompt caching can match the prefix, send anything that can wait through the batch route at around half price.","keyIdea":"Put every stable block at the front of a request so prompt caching can match the prefix, send anything that can wait through the batch route at around half price, and cut what you send and what you ask for before you consider a cheaper model."},"body":{"format":"markdown","text":"## Prompt caching\n\nIf you send the same long block of text repeatedly (a system prompt with detailed instructions, a policy document, a code file, a knowledge base) the provider can store its processed form and charge a fraction to reuse it. Discounts are typically 50% to 90% on the cached portion, and some providers apply it automatically while others need a marker in the request.\n\nThe mechanism decides how you use it. The cache matches on an **exact prefix**: the request has to begin with byte-identical text. So the rule is to put everything stable at the front and everything variable at the back.\n\n```\n[ long instructions        ]  <- identical every time, cached\n[ reference document       ]  <- identical every time, cached\n[ today's date, user's name]  <- CHANGES: breaks the cache from here on\n[ the question             ]\n\nmove the date and the name to the END, after the question,\nand the whole block above stays cached\n```\n\nThat single reordering is worth a large fraction of the bill on any workload with a fixed preamble, and it is the most common cache mistake: one variable field near the top invalidates everything after it.\n\nCaches expire: commonly a few minutes of inactivity, extendable at a price on some providers. So caching pays when requests come in bursts and does nothing for one request an hour.\n\n> Figure: Order the request so the cache can match. Layers, top to bottom: Long instructions (Byte-identical every time. Cached.); Reference document (Byte-identical every time. Cached.); The question (Varies. Everything from here down is charged in full, which is correct and unavoidable.); Today's date, the user's name (Move these below the question. One variable field near the top invalidates every cached block after it.). The cache matches on an exact prefix, so every stable block goes at the front and everything variable at the back. Discounts run from a half to nine tenths on the matched portion, and the commonest mistake is a date sitting two lines from the top.\n\n## Batch processing\n\nMost providers offer a batch route: you upload many requests as a file, they are processed within a stated window (usually up to 24 hours) and you collect the results. The discount is generally around 50%.\n\nThis is the right route for anything that does not need an answer now: classifying an archive, generating descriptions for a catalogue, summarising last month's tickets, translating a documentation set. Half price for patience.\n\n```\n50,000 items, live:   $458\n50,000 items, batch:  $229\n```\n\nTwo practical notes. Results come back unordered, so include your own identifier with each request. And the window is a maximum, not a promise of speed: most batches finish much sooner, but you cannot rely on it, so do not put a batch job in front of a deadline.\n\n## The discounts that need nothing from the provider\n\n**Stop resending the conversation.** In a chat front-end, cap the history at the last ten messages. Most of the cost of a long conversation is the earlier turns being re-read every time.\n\n**Ask for shorter answers.** Output is the expensive side. A model told to answer in 100 words instead of 500 costs a fifth as much for the generation, and for most questions the 100-word version is the one you wanted.\n\n**Send less.** People attach a whole document when a section would do. Extraction with `pdftotext` and a few lines of slicing costs nothing and can cut a request by 90%.\n\n**Turn thinking off when it does not help.** Covered in module one; it belongs on this list because it is the biggest single lever.\n\n**Deduplicate.** If the same question is asked repeatedly by different people, store answers in a table keyed by the question. A dictionary lookup costs nothing and often serves a third of the traffic in customer-facing work.\n\n## What none of this fixes\n\nTwo costs are immune to all of it. A model that has to read a long document to answer a global question about it (is anything inconsistent, what is the overall argument) must see the whole thing, and no amount of caching, batching or routing avoids paying for that once.\n\nAnd an answer that has to be right cannot be made cheaper by being generated more cheaply. Where the cost of an error dominates, the correct spending decision is usually to pay for the better model and the review step, and to look for savings on the routine work that surrounds it instead.\n\n## The ordering that matters\n\nIf you want to cut a bill, do these in this order, because the first three cost nothing and the last one costs quality:\n\n1. Send less and ask for less.\n2. Cache the stable prefix, batch what can wait.\n3. Route easy items to a smaller model.\n4. Only then consider accepting worse output.\n\nMost people go straight to step four by switching to a cheaper model everywhere, discover the quality drop, and conclude the work cannot be done cheaply. The first two steps are usually worth more than the model change, and they cost you nothing at all."},"check":{"question":"A workload puts the current date at the top of an otherwise identical 8,000-token preamble, and prompt caching saves almost nothing. Why?","options":[{"text":"The preamble is too long to cache, since providers cap the size of a cacheable block well below 8,000 tokens.","correct":false,"tempting":"Invents a low size cap; long stable preambles are exactly what caching is designed for, with minimums rather than tight maximums."},{"text":"Caching only applies to system messages, and the date placed at the top pushed the instructions into the user message.","correct":false,"tempting":"Adds a role restriction that does not exist, when the match is on the leading text regardless of which role carries it."},{"text":"Dates are treated as volatile content and excluded from caching by the provider's content rules.","correct":false,"tempting":"Attributes to a content rule what is a plain string comparison; the cache has no notion of what a date means."},{"text":"The cache matches on an exact prefix, so a value that changes daily invalidates everything after it.","correct":true}]},"prev":{"slug":"estimating-a-job","title":"Costing a job before you run it","url":"https://addaly.com/learn/choosing-your-tools/estimating-a-job"},"next":{"slug":"routing-by-difficulty","title":"Spending money where it matters","url":"https://addaly.com/learn/choosing-your-tools/routing-by-difficulty"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The two discounts almost nobody uses', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/caching-and-batching (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The two discounts almost nobody uses","courseTitle":"Choosing and Using the Tools","module":{"index":4,"slug":"what-it-costs","title":"What it actually costs"},"answer":null,"keyIdea":"Put every stable block at the front of a request so prompt caching can match the prefix, send anything that can wait through the batch route at around half price, and cut what you send and what you ask for before you consider a cheaper model.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Order the request so the cache can match","layers":[{"label":"Long instructions","detail":"Byte-identical every time. Cached."},{"label":"Reference document","detail":"Byte-identical every time. Cached."},{"label":"The question","detail":"Varies. Everything from here down is charged in full, which is correct and unavoidable."},{"label":"Today’s date, the user’s name","detail":"Move these below the question. One variable field near the top invalidates every cached block after it."}],"caption":"The cache matches on an exact prefix, so every stable block goes at the front and everything variable at the back. Discounts run from a half to nine tenths on the matched portion, and the commonest mistake is a date sitting two lines from the top."},"alt":"Figure: Order the request so the cache can match. Layers, top to bottom: Long instructions (Byte-identical every time. Cached.); Reference document (Byte-identical every time. Cached.); The question (Varies. Everything from here down is charged in full, which is correct and unavoidable.); Today’s date, the user’s name (Move these below the question. One variable field near the top invalidates every cached block after it.). The cache matches on an exact prefix, so every stable block goes at the front and everything variable at the back. Discounts run from a half to nine tenths on the matched portion, and the commonest mistake is a date sitting two lines from the top."}},"blocks":[{"t":"h2","id":"prompt-caching","runs":[{"text":"Prompt caching"}]},{"t":"p","runs":[{"text":"If you send the same long block of text repeatedly (a system prompt with detailed instructions, a policy document, a code file, a knowledge base) the provider can store its processed form and charge a fraction to reuse it. Discounts are typically 50% to 90% on the cached portion, and some providers apply it automatically while others need a marker in the request."}]},{"t":"p","runs":[{"text":"The mechanism decides how you use it. The cache matches on an "},{"text":"exact prefix","b":true},{"text":": the request has to begin with byte-identical text. So the rule is to put everything stable at the front and everything variable at the back."}]},{"t":"code","lang":"","label":"Text","text":"[ long instructions        ]  <- identical every time, cached\n[ reference document       ]  <- identical every time, cached\n[ today's date, user's name]  <- CHANGES: breaks the cache from here on\n[ the question             ]\n\nmove the date and the name to the END, after the question,\nand the whole block above stays cached"},{"t":"p","runs":[{"text":"That single reordering is worth a large fraction of the bill on any workload with a fixed preamble, and it is the most common cache mistake: one variable field near the top invalidates everything after it."}]},{"t":"p","runs":[{"text":"Caches expire: commonly a few minutes of inactivity, extendable at a price on some providers. So caching pays when requests come in bursts and does nothing for one request an hour."}]},{"t":"h2","id":"batch-processing","runs":[{"text":"Batch processing"}]},{"t":"p","runs":[{"text":"Most providers offer a batch route: you upload many requests as a file, they are processed within a stated window (usually up to 24 hours) and you collect the results. The discount is generally around 50%."}]},{"t":"p","runs":[{"text":"This is the right route for anything that does not need an answer now: classifying an archive, generating descriptions for a catalogue, summarising last month’s tickets, translating a documentation set. Half price for patience."}]},{"t":"code","lang":"","label":"Text","text":"50,000 items, live:   $458\n50,000 items, batch:  $229"},{"t":"p","runs":[{"text":"Two practical notes. Results come back unordered, so include your own identifier with each request. And the window is a maximum, not a promise of speed: most batches finish much sooner, but you cannot rely on it, so do not put a batch job in front of a deadline."}]},{"t":"h2","id":"the-discounts-that-need-nothing-from-the-provider","runs":[{"text":"The discounts that need nothing from the provider"}]},{"t":"p","runs":[{"text":"Stop resending the conversation.","b":true},{"text":" In a chat front-end, cap the history at the last ten messages. Most of the cost of a long conversation is the earlier turns being re-read every time."}]},{"t":"p","runs":[{"text":"Ask for shorter answers.","b":true},{"text":" Output is the expensive side. A model told to answer in 100 words instead of 500 costs a fifth as much for the generation, and for most questions the 100-word version is the one you wanted."}]},{"t":"p","runs":[{"text":"Send less.","b":true},{"text":" People attach a whole document when a section would do. Extraction with "},{"text":"pdftotext","code":true},{"text":" and a few lines of slicing costs nothing and can cut a request by 90%."}]},{"t":"p","runs":[{"text":"Turn thinking off when it does not help.","b":true},{"text":" Covered in module one; it belongs on this list because it is the biggest single lever."}]},{"t":"p","runs":[{"text":"Deduplicate.","b":true},{"text":" If the same question is asked repeatedly by different people, store answers in a table keyed by the question. A dictionary lookup costs nothing and often serves a third of the traffic in customer-facing work."}]},{"t":"h2","id":"what-none-of-this-fixes","runs":[{"text":"What none of this fixes"}]},{"t":"p","runs":[{"text":"Two costs are immune to all of it. A model that has to read a long document to answer a global question about it (is anything inconsistent, what is the overall argument) must see the whole thing, and no amount of caching, batching or routing avoids paying for that once."}]},{"t":"p","runs":[{"text":"And an answer that has to be right cannot be made cheaper by being generated more cheaply. Where the cost of an error dominates, the correct spending decision is usually to pay for the better model and the review step, and to look for savings on the routine work that surrounds it instead."}]},{"t":"h2","id":"the-ordering-that-matters","runs":[{"text":"The ordering that matters"}]},{"t":"p","runs":[{"text":"If you want to cut a bill, do these in this order, because the first three cost nothing and the last one costs quality:"}]},{"t":"ol","start":1,"items":[[{"text":"Send less and ask for less."}],[{"text":"Cache the stable prefix, batch what can wait."}],[{"text":"Route easy items to a smaller model."}],[{"text":"Only then consider accepting worse output."}]]},{"t":"p","runs":[{"text":"Most people go straight to step four by switching to a cheaper model everywhere, discover the quality drop, and conclude the work cannot be done cheaply. The first two steps are usually worth more than the model change, and they cost you nothing at all."}]}],"check":{"question":[{"text":"A workload puts the current date at the top of an otherwise identical 8,000-token preamble, and prompt caching saves almost nothing. Why?"}],"options":[{"text":[{"text":"The preamble is too long to cache, since providers cap the size of a cacheable block well below 8,000 tokens."}],"correct":false,"tempting":[{"text":"Invents a low size cap; long stable preambles are exactly what caching is designed for, with minimums rather than tight maximums."}]},{"text":[{"text":"Caching only applies to system messages, and the date placed at the top pushed the instructions into the user message."}],"correct":false,"tempting":[{"text":"Adds a role restriction that does not exist, when the match is on the leading text regardless of which role carries it."}]},{"text":[{"text":"Dates are treated as volatile content and excluded from caching by the provider’s content rules."}],"correct":false,"tempting":[{"text":"Attributes to a content rule what is a plain string comparison; the cache has no notion of what a date means."}]},{"text":[{"text":"The cache matches on an exact prefix, so a value that changes daily invalidates everything after it."}],"correct":true,"tempting":null}],"order":[1,2,3,0],"why":null},"prev":{"slug":"estimating-a-job","title":"Costing a job before you run it","minutes":8},"next":{"slug":"routing-by-difficulty","title":"Spending money where it matters","minutes":8}}}