{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"the-hour-long-audition","title":"The audition: settling it for about a dollar","url":"https://addaly.com/learn/running-models-yourself/the-hour-long-audition","position":7,"of":82,"minutes":9,"answer":"Test the weights before the hardware: thirty real items graded blind against a hosted copy of the exact open model answers the question a GPU purchase cannot, for under a dollar.","keyIdea":"Test the weights before the hardware: thirty real items graded blind against a hosted copy of the exact open model answers the question a GPU purchase cannot, for under a dollar."},"body":{"format":"markdown","text":"## The mistake this prevents\n\nThe usual order is: buy hardware, install a stack, download a model, discover it is not good enough, and conclude that local models do not work. The cost of that sequence is a weekend and possibly a graphics card.\n\nThe fix is to separate the two questions and answer the harder one first. *Are these weights good enough for my task?* is a property of the model. *Can I run them?* is a property of my machine. The first question can be answered in an hour, on any machine, for roughly the price of a coffee, because almost every open model is also served by several hosted providers, running the identical weights.\n\nIf the model fails the audition, no amount of hardware helps. If it passes, you now have a target and can shop for the cheapest way to run exactly those weights.\n\n## Build the set first\n\nTwenty to thirty real items from your actual work. Not invented examples: invented examples are always easier than reality, in a specific way: they are shorter, cleaner, better punctuated, and free of the internal jargon that breaks models.\n\n> Figure: One hour, about one dollar, and the question is settled. Timeline: 0 to 20 min, Collect thirty real items from the actual work. Half the common case, several awkward ones, and two or three you expect to fail. Write the acceptance criteria before running anything.; 20 to 30 min, Run them through a hosted copy of the exact open weights, with the sampler pinned to something you can reproduce. Thirty prompts of 800 tokens is under a cent of input.; 30 to 50 min, Strip the model names, shuffle the outputs, and grade blind on three points: usable, usable after a small edit, not usable.; 50 to 60 min, Re-grade five items you scored earlier without looking at your first answer. If more than one disagrees, your criteria are too vague to compare models with.; Afterwards, A clear pass names the exact weights to shop for. A clear failure means no graphics card on earth will change the result.. This answers whether the weights are good enough, which is a property of the model. Whether you can run them is a property of your machine, and it is the easier question. Doing them in the other order is how people end up with a graphics card and a disappointment.\n\nCover the shape of the work deliberately:\n\n- The common case, about half the items.\n- The awkward cases you know about: long inputs, mixed languages, tables pasted as text, abbreviations, typos.\n- Two or three items you expect the model to fail. If nothing fails, your set is too easy to distinguish between models.\n\nWrite the expected output, or at least the acceptance criteria, before you run anything. Grading after the fact against no standard is how everything comes out looking acceptable.\n\n## Run the identical weights\n\nOpen models are served through aggregators and through the labs' own endpoints, typically at a fraction of a cent per thousand tokens. Thirty prompts of 800 tokens each is under a cent of input; even at generous output lengths the whole exercise is well under a dollar.\n\nTwo things matter for the comparison to be valid:\n\n1. **Check the exact model identifier**, including the size and the instruct suffix, and note whether the provider is serving it quantised. Many hosted providers serve fp8 or 4-bit versions without saying so on the pricing page, which is fine but changes what you are measuring.\n2. **Set the sampler to something you can reproduce**: temperature 0.7, top-p 0.95, or temperature 0 for anything deterministic. If you leave the provider's defaults you cannot compare against your local run later.\n\n```bash\ncurl https://api.example-provider.com/v1/chat/completions \\\n  -H \"Authorization: Bearer $KEY\" -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"qwen/qwen3-8b\",\"temperature\":0,\n       \"messages\":[{\"role\":\"user\",\"content\":\"...\"}]}'\n```\n\nRun the whole set through a short script and save the outputs to a file with the prompt beside each one.\n\n## Grade blind, and grade twice\n\nPut two or three candidate models through the same set. Then strip the model names, shuffle, and grade the outputs without knowing which produced which. This is not ceremony. Knowing which model you are looking at moves scores by a startling amount, in whichever direction you already believed.\n\nScore each item on a three-point scale (usable as is, usable after a small edit, not usable) rather than a ten-point one. Ten-point scales produce fake precision and take four times as long.\n\nThen re-grade five items you scored earlier in the session, without looking at your first answer. If your two scores disagree on more than one of the five, your criteria are too vague to compare models with, and tightening them is the most valuable thing you can do next.\n\n## Read the result correctly\n\n**Clear failure.** More than about a third not usable. Stop. A bigger card will not change this. Either the task needs a frontier model, or it needs decomposition into smaller steps, or it needs retrieval rather than a model that knows things.\n\n**Clear pass.** Nearly everything usable, and the failures are ones you can filter or route around. You now know the exact weights to run, and the rest of this course is about running them.\n\n**The awkward middle.** Half usable. This is the common outcome and it is where the money is. Before concluding anything, try in this order: fix the prompt, try the next size up in the same family, try a task-specialised model, and try a newer model in the same size class. Each takes another twenty minutes and any of them can move a middling result into a clear pass.\n\n## Keep the set\n\nThat file of thirty items with expected outputs is now the most valuable artefact you own for this project. It tells you whether a quantised copy still works, whether the new release is better, whether the sampler change helped, and whether the fine-tune you spent a weekend on actually improved anything. Every later lesson in this course assumes you have it."},"check":{"question":"An audition of a hosted 8B gives 60% usable output. The team's plan is to buy a 24 GB card and run the same 8B locally, expecting better results because there will be no provider-side quantisation. What is wrong with the reasoning?","options":[{"text":"Local inference has no rate limit, so the real gain will come from retrying failed items rather than from precision.","correct":false,"tempting":"Substitutes a throughput benefit for a quality one and quietly assumes retries fix systematic failures, which they do not."},{"text":"Removing quantisation could recover a few points at most; a 40% failure rate is a property of the weights, so the money should go to prompt work, a larger or specialised model, or task decomposition first.","correct":true},{"text":"The hosted result is invalid because provider system prompts contaminate the comparison, so the audition needs redoing locally before any conclusion.","correct":false,"tempting":"Treats a manageable variable as fatal; providers serving raw completions do not inject prompts, and this would make pre-purchase testing impossible in principle."},{"text":"A 24 GB card cannot run an 8B at full precision with useful context, so the plan fails on memory before it fails on quality.","correct":false,"tempting":"Is arithmetically wrong (an 8B in bf16 is about 16 GB and leaves room for context) and answers a fit question when the problem is a capability one."}]},"prev":{"slug":"base-instruct-and-reasoning-variants","title":"Base, instruct and reasoning: three different products","url":"https://addaly.com/learn/running-models-yourself/base-instruct-and-reasoning-variants"},"next":{"slug":"what-it-will-never-do","title":"The ceiling the weights impose","url":"https://addaly.com/learn/running-models-yourself/what-it-will-never-do"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The audition: settling it for about a dollar', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/the-hour-long-audition (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The audition: settling it for about a dollar","courseTitle":"Running Models Yourself","module":{"index":1,"slug":"deciding-to-run-it-yourself","title":"Deciding to run it yourself"},"answer":null,"keyIdea":"Test the weights before the hardware: thirty real items graded blind against a hosted copy of the exact open model answers the question a GPU purchase cannot, for under a dollar.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"One hour, about one dollar, and the question is settled","events":[{"when":"0 to 20 min","what":"Collect thirty real items from the actual work. Half the common case, several awkward ones, and two or three you expect to fail. Write the acceptance criteria before running anything."},{"when":"20 to 30 min","what":"Run them through a hosted copy of the exact open weights, with the sampler pinned to something you can reproduce. Thirty prompts of 800 tokens is under a cent of input."},{"when":"30 to 50 min","what":"Strip the model names, shuffle the outputs, and grade blind on three points: usable, usable after a small edit, not usable."},{"when":"50 to 60 min","what":"Re-grade five items you scored earlier without looking at your first answer. If more than one disagrees, your criteria are too vague to compare models with."},{"when":"Afterwards","what":"A clear pass names the exact weights to shop for. A clear failure means no graphics card on earth will change the result."}],"caption":"This answers whether the weights are good enough, which is a property of the model. Whether you can run them is a property of your machine, and it is the easier question. Doing them in the other order is how people end up with a graphics card and a disappointment."},"alt":"Figure: One hour, about one dollar, and the question is settled. Timeline: 0 to 20 min, Collect thirty real items from the actual work. Half the common case, several awkward ones, and two or three you expect to fail. Write the acceptance criteria before running anything.; 20 to 30 min, Run them through a hosted copy of the exact open weights, with the sampler pinned to something you can reproduce. Thirty prompts of 800 tokens is under a cent of input.; 30 to 50 min, Strip the model names, shuffle the outputs, and grade blind on three points: usable, usable after a small edit, not usable.; 50 to 60 min, Re-grade five items you scored earlier without looking at your first answer. If more than one disagrees, your criteria are too vague to compare models with.; Afterwards, A clear pass names the exact weights to shop for. A clear failure means no graphics card on earth will change the result.. This answers whether the weights are good enough, which is a property of the model. Whether you can run them is a property of your machine, and it is the easier question. Doing them in the other order is how people end up with a graphics card and a disappointment."}},"blocks":[{"t":"h2","id":"the-mistake-this-prevents","runs":[{"text":"The mistake this prevents"}]},{"t":"p","runs":[{"text":"The usual order is: buy hardware, install a stack, download a model, discover it is not good enough, and conclude that local models do not work. The cost of that sequence is a weekend and possibly a graphics card."}]},{"t":"p","runs":[{"text":"The fix is to separate the two questions and answer the harder one first. "},{"text":"Are these weights good enough for my task?","i":true},{"text":" is a property of the model. "},{"text":"Can I run them?","i":true},{"text":" is a property of my machine. The first question can be answered in an hour, on any machine, for roughly the price of a coffee, because almost every open model is also served by several hosted providers, running the identical weights."}]},{"t":"p","runs":[{"text":"If the model fails the audition, no amount of hardware helps. If it passes, you now have a target and can shop for the cheapest way to run exactly those weights."}]},{"t":"h2","id":"build-the-set-first","runs":[{"text":"Build the set first"}]},{"t":"p","runs":[{"text":"Twenty to thirty real items from your actual work. Not invented examples: invented examples are always easier than reality, in a specific way: they are shorter, cleaner, better punctuated, and free of the internal jargon that breaks models."}]},{"t":"p","runs":[{"text":"Cover the shape of the work deliberately:"}]},{"t":"ul","items":[[{"text":"The common case, about half the items."}],[{"text":"The awkward cases you know about: long inputs, mixed languages, tables pasted as text, abbreviations, typos."}],[{"text":"Two or three items you expect the model to fail. If nothing fails, your set is too easy to distinguish between models."}]]},{"t":"p","runs":[{"text":"Write the expected output, or at least the acceptance criteria, before you run anything. Grading after the fact against no standard is how everything comes out looking acceptable."}]},{"t":"h2","id":"run-the-identical-weights","runs":[{"text":"Run the identical weights"}]},{"t":"p","runs":[{"text":"Open models are served through aggregators and through the labs’ own endpoints, typically at a fraction of a cent per thousand tokens. Thirty prompts of 800 tokens each is under a cent of input; even at generous output lengths the whole exercise is well under a dollar."}]},{"t":"p","runs":[{"text":"Two things matter for the comparison to be valid:"}]},{"t":"ol","start":1,"items":[[{"text":"Check the exact model identifier","b":true},{"text":", including the size and the instruct suffix, and note whether the provider is serving it quantised. Many hosted providers serve fp8 or 4-bit versions without saying so on the pricing page, which is fine but changes what you are measuring."}],[{"text":"Set the sampler to something you can reproduce","b":true},{"text":": temperature 0.7, top-p 0.95, or temperature 0 for anything deterministic. If you leave the provider’s defaults you cannot compare against your local run later."}]]},{"t":"code","lang":"bash","label":"Bash","text":"curl https://api.example-provider.com/v1/chat/completions \\\n  -H \"Authorization: Bearer $KEY\" -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"qwen/qwen3-8b\",\"temperature\":0,\n       \"messages\":[{\"role\":\"user\",\"content\":\"...\"}]}'"},{"t":"p","runs":[{"text":"Run the whole set through a short script and save the outputs to a file with the prompt beside each one."}]},{"t":"h2","id":"grade-blind-and-grade-twice","runs":[{"text":"Grade blind, and grade twice"}]},{"t":"p","runs":[{"text":"Put two or three candidate models through the same set. Then strip the model names, shuffle, and grade the outputs without knowing which produced which. This is not ceremony. Knowing which model you are looking at moves scores by a startling amount, in whichever direction you already believed."}]},{"t":"p","runs":[{"text":"Score each item on a three-point scale (usable as is, usable after a small edit, not usable) rather than a ten-point one. Ten-point scales produce fake precision and take four times as long."}]},{"t":"p","runs":[{"text":"Then re-grade five items you scored earlier in the session, without looking at your first answer. If your two scores disagree on more than one of the five, your criteria are too vague to compare models with, and tightening them is the most valuable thing you can do next."}]},{"t":"h2","id":"read-the-result-correctly","runs":[{"text":"Read the result correctly"}]},{"t":"p","runs":[{"text":"Clear failure.","b":true},{"text":" More than about a third not usable. Stop. A bigger card will not change this. Either the task needs a frontier model, or it needs decomposition into smaller steps, or it needs retrieval rather than a model that knows things."}]},{"t":"p","runs":[{"text":"Clear pass.","b":true},{"text":" Nearly everything usable, and the failures are ones you can filter or route around. You now know the exact weights to run, and the rest of this course is about running them."}]},{"t":"p","runs":[{"text":"The awkward middle.","b":true},{"text":" Half usable. This is the common outcome and it is where the money is. Before concluding anything, try in this order: fix the prompt, try the next size up in the same family, try a task-specialised model, and try a newer model in the same size class. Each takes another twenty minutes and any of them can move a middling result into a clear pass."}]},{"t":"h2","id":"keep-the-set","runs":[{"text":"Keep the set"}]},{"t":"p","runs":[{"text":"That file of thirty items with expected outputs is now the most valuable artefact you own for this project. It tells you whether a quantised copy still works, whether the new release is better, whether the sampler change helped, and whether the fine-tune you spent a weekend on actually improved anything. Every later lesson in this course assumes you have it."}]}],"check":{"question":[{"text":"An audition of a hosted 8B gives 60% usable output. The team’s plan is to buy a 24 GB card and run the same 8B locally, expecting better results because there will be no provider-side quantisation. What is wrong with the reasoning?"}],"options":[{"text":[{"text":"Local inference has no rate limit, so the real gain will come from retrying failed items rather than from precision."}],"correct":false,"tempting":[{"text":"Substitutes a throughput benefit for a quality one and quietly assumes retries fix systematic failures, which they do not."}]},{"text":[{"text":"Removing quantisation could recover a few points at most; a 40% failure rate is a property of the weights, so the money should go to prompt work, a larger or specialised model, or task decomposition first."}],"correct":true,"tempting":null},{"text":[{"text":"The hosted result is invalid because provider system prompts contaminate the comparison, so the audition needs redoing locally before any conclusion."}],"correct":false,"tempting":[{"text":"Treats a manageable variable as fatal; providers serving raw completions do not inject prompts, and this would make pre-purchase testing impossible in principle."}]},{"text":[{"text":"A 24 GB card cannot run an 8B at full precision with useful context, so the plan fails on memory before it fails on quality."}],"correct":false,"tempting":[{"text":"Is arithmetically wrong (an 8B in bf16 is about 16 GB and leaves room for context) and answers a fit question when the problem is a capability one."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"base-instruct-and-reasoning-variants","title":"Base, instruct and reasoning: three different products","minutes":8},"next":{"slug":"what-it-will-never-do","title":"The ceiling the weights impose","minutes":8}}}