A rubric you can apply in ten seconds

Why a score beats an impression

“That one felt better” is not portable. You cannot compare it to what you felt six months ago, you cannot explain it to a colleague, and you cannot tell whether it was the answer or your mood. A number, even a crude one, fixes all three.

The rubric that works for personal tool choice is deliberately blunt. Three points per answer:

• 2: I would use this as it stands, or with a trivial edit.
• 1: Useful, but I would have to fix something before using it.
• 0: Wrong, or I would rewrite it from scratch.

Twelve prompts, twenty-four points available, one number per tool. It takes ten seconds an answer and it is enough.

The pre-commitment that makes it work

The rubric is not the hard part. The hard part is that you decided what “good” meant before you saw any answers, on the good: line in your prompt file. Without that, scoring drifts: you read a fluent answer, it impresses you, and the criteria quietly rearrange themselves to match what you just read.

If you only take one habit from this module, take that one. Written criteria first, answers second.

Four ways scoring goes wrong

Length. Longer answers score higher, from human and machine judges alike, largely regardless of merit. When two answers are both correct, the shorter is better; you have to read it.

Formatting. Headings, bold text and bullet points read as thorough. They are a house style set by a system prompt, not evidence of quality. A tool that returns three plain sentences containing everything you needed has beaten the one that returned a formatted page containing the same three sentences.

Confidence. A decisive wrong answer scores above a hedged right one, unless you are watching for it. Your known-answer probes are the correction, which is why they are in the file.

Agreement with you. An answer that confirms what you already thought feels correct. On anything where you have a prior view, force yourself to ask what you would say about this answer if it contradicted you.

When two things matter separately

Sometimes one score hides the decision. If both correctness and tone matter to you and the tools split (one accurate and stiff, one warm and occasionally wrong) score two columns rather than averaging them into a number that describes neither.

Two columns is the maximum. Beyond that you are building an evaluation framework rather than choosing a tool, and the extra columns get filled in carelessly by the eighth prompt.

Scoring answers you are not qualified to judge

There is one case where this rubric fails honestly, and pretending otherwise would be useless. If you cannot tell whether an answer is correct (a legal question outside your area, code in a language you do not read, a translation into a language you only half speak) then a score of 2 means “this looked good to me”, which is exactly the judgement these tools are best at defeating.

Two honest options. Either mark it ? and get one person who can judge it to look at three answers, or replace that prompt with one where you can verify the answer. Do not score it and pretend the number means something; a false 2 on the prompt you understand least will quietly dominate a close comparison.

Recording the result

run 2026-03-14   12 prompts, blind, one sitting
  toolA (gpt-4o-mini-2024-07-18)     19/24   fast, terse, weak on Hindi register
  toolB (claude-sonnet-4-5)          21/24   best on the long contract
  local (qwen3-8b, Q4_K_M, laptop)   14/24   fine on 1-6, lost on 7-9
decision: toolB paid, toolA free tier as second opinion, local for
          anything with client names in it
revisit: 2026-09-14

Six lines. That is a decision record, and it does three things an impression cannot: it tells your future self what you actually observed, it names the exact versions so you can tell later whether the tool changed or you did, and it states the decision so you stop relitigating it every time somebody posts a benchmark.

The honest limit of all this

Twelve prompts scored zero to two by one person is not a rigorous evaluation and does not pretend to be. It cannot estimate an error rate, it cannot detect a five per cent difference, and a different person with different work would reasonably reach a different answer.

It does not need to do any of that. It needs to be better than choosing on the basis of a headline, and it clears that bar by an enormous margin, at a cost of one hour twice a year.