The audition: settling it for about a dollar

The mistake this prevents

The usual order is: buy hardware, install a stack, download a model, discover it is not good enough, and conclude that local models do not work. The cost of that sequence is a weekend and possibly a graphics card.

The fix is to separate the two questions and answer the harder one first. Are these weights good enough for my task? is a property of the model. Can I run them? is a property of my machine. The first question can be answered in an hour, on any machine, for roughly the price of a coffee, because almost every open model is also served by several hosted providers, running the identical weights.

If the model fails the audition, no amount of hardware helps. If it passes, you now have a target and can shop for the cheapest way to run exactly those weights.

Build the set first

Twenty to thirty real items from your actual work. Not invented examples: invented examples are always easier than reality, in a specific way: they are shorter, cleaner, better punctuated, and free of the internal jargon that breaks models.

Cover the shape of the work deliberately:

• The common case, about half the items.
• The awkward cases you know about: long inputs, mixed languages, tables pasted as text, abbreviations, typos.
• Two or three items you expect the model to fail. If nothing fails, your set is too easy to distinguish between models.

Write the expected output, or at least the acceptance criteria, before you run anything. Grading after the fact against no standard is how everything comes out looking acceptable.

Run the identical weights

Open models are served through aggregators and through the labs’ own endpoints, typically at a fraction of a cent per thousand tokens. Thirty prompts of 800 tokens each is under a cent of input; even at generous output lengths the whole exercise is well under a dollar.

Two things matter for the comparison to be valid:

1. Check the exact model identifier, including the size and the instruct suffix, and note whether the provider is serving it quantised. Many hosted providers serve fp8 or 4-bit versions without saying so on the pricing page, which is fine but changes what you are measuring.
2. Set the sampler to something you can reproduce: temperature 0.7, top-p 0.95, or temperature 0 for anything deterministic. If you leave the provider’s defaults you cannot compare against your local run later.

curl https://api.example-provider.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"qwen/qwen3-8b","temperature":0,
       "messages":[{"role":"user","content":"..."}]}'

Run the whole set through a short script and save the outputs to a file with the prompt beside each one.

Grade blind, and grade twice

Put two or three candidate models through the same set. Then strip the model names, shuffle, and grade the outputs without knowing which produced which. This is not ceremony. Knowing which model you are looking at moves scores by a startling amount, in whichever direction you already believed.

Score each item on a three-point scale (usable as is, usable after a small edit, not usable) rather than a ten-point one. Ten-point scales produce fake precision and take four times as long.

Then re-grade five items you scored earlier in the session, without looking at your first answer. If your two scores disagree on more than one of the five, your criteria are too vague to compare models with, and tightening them is the most valuable thing you can do next.

Read the result correctly

Clear failure. More than about a third not usable. Stop. A bigger card will not change this. Either the task needs a frontier model, or it needs decomposition into smaller steps, or it needs retrieval rather than a model that knows things.

Clear pass. Nearly everything usable, and the failures are ones you can filter or route around. You now know the exact weights to run, and the rest of this course is about running them.

The awkward middle. Half usable. This is the common outcome and it is where the money is. Before concluding anything, try in this order: fix the prompt, try the next size up in the same family, try a task-specialised model, and try a newer model in the same size class. Each takes another twenty minutes and any of them can move a middling result into a clear pass.

Keep the set

That file of thirty items with expected outputs is now the most valuable artefact you own for this project. It tells you whether a quantised copy still works, whether the new release is better, whether the sampler change helped, and whether the fine-tune you spent a weekend on actually improved anything. Every later lesson in this course assumes you have it.