{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"scoring-without-fooling-yourself","title":"A rubric you can apply in ten seconds","url":"https://addaly.com/learn/choosing-your-tools/scoring-without-fooling-yourself","position":16,"of":76,"minutes":7,"answer":"Score each answer 0, 1 or 2 against criteria written before you saw any answers, and watch for the four biases (length, formatting, confidence and agreement) that inflate scores without improving usefulness.","keyIdea":"Score each answer 0, 1 or 2 against criteria written before you saw any answers, and watch for the four biases (length, formatting, confidence and agreement) that inflate scores without improving usefulness."},"body":{"format":"markdown","text":"## Why a score beats an impression\n\n\"That one felt better\" is not portable. You cannot compare it to what you felt six months ago, you cannot explain it to a colleague, and you cannot tell whether it was the answer or your mood. A number, even a crude one, fixes all three.\n\nThe rubric that works for personal tool choice is deliberately blunt. Three points per answer:\n\n- **2**: I would use this as it stands, or with a trivial edit.\n- **1**: Useful, but I would have to fix something before using it.\n- **0**: Wrong, or I would rewrite it from scratch.\n\nTwelve prompts, twenty-four points available, one number per tool. It takes ten seconds an answer and it is enough.\n\n## The pre-commitment that makes it work\n\nThe rubric is not the hard part. The hard part is that you decided what \"good\" meant before you saw any answers, on the `good:` line in your prompt file. Without that, scoring drifts: you read a fluent answer, it impresses you, and the criteria quietly rearrange themselves to match what you just read.\n\nIf you only take one habit from this module, take that one. Written criteria first, answers second.\n\n## Four ways scoring goes wrong\n\n**Length.** Longer answers score higher, from human and machine judges alike, largely regardless of merit. When two answers are both correct, the shorter is better; you have to read it.\n\n**Formatting.** Headings, bold text and bullet points read as thorough. They are a house style set by a system prompt, not evidence of quality. A tool that returns three plain sentences containing everything you needed has beaten the one that returned a formatted page containing the same three sentences.\n\n**Confidence.** A decisive wrong answer scores above a hedged right one, unless you are watching for it. Your known-answer probes are the correction, which is why they are in the file.\n\n**Agreement with you.** An answer that confirms what you already thought feels correct. On anything where you have a prior view, force yourself to ask what you would say about this answer if it contradicted you.\n\n## When two things matter separately\n\nSometimes one score hides the decision. If both correctness and tone matter to you and the tools split (one accurate and stiff, one warm and occasionally wrong) score two columns rather than averaging them into a number that describes neither.\n\nTwo columns is the maximum. Beyond that you are building an evaluation framework rather than choosing a tool, and the extra columns get filled in carelessly by the eighth prompt.\n\n## Scoring answers you are not qualified to judge\n\nThere is one case where this rubric fails honestly, and pretending otherwise would be useless. If you cannot tell whether an answer is correct (a legal question outside your area, code in a language you do not read, a translation into a language you only half speak) then a score of 2 means \"this looked good to me\", which is exactly the judgement these tools are best at defeating.\n\nTwo honest options. Either mark it `?` and get one person who can judge it to look at three answers, or replace that prompt with one where you can verify the answer. Do not score it and pretend the number means something; a false 2 on the prompt you understand least will quietly dominate a close comparison.\n\n## Recording the result\n\n```\nrun 2026-03-14   12 prompts, blind, one sitting\n  toolA (gpt-4o-mini-2024-07-18)     19/24   fast, terse, weak on Hindi register\n  toolB (claude-sonnet-4-5)          21/24   best on the long contract\n  local (qwen3-8b, Q4_K_M, laptop)   14/24   fine on 1-6, lost on 7-9\ndecision: toolB paid, toolA free tier as second opinion, local for\n          anything with client names in it\nrevisit: 2026-09-14\n```\n\nSix lines. That is a decision record, and it does three things an impression cannot: it tells your future self what you actually observed, it names the exact versions so you can tell later whether the tool changed or you did, and it states the decision so you stop relitigating it every time somebody posts a benchmark.\n\n## The honest limit of all this\n\nTwelve prompts scored zero to two by one person is not a rigorous evaluation and does not pretend to be. It cannot estimate an error rate, it cannot detect a five per cent difference, and a different person with different work would reasonably reach a different answer.\n\nIt does not need to do any of that. It needs to be better than choosing on the basis of a headline, and it clears that bar by an enormous margin, at a cost of one hour twice a year."},"check":{"question":"Two tools tie on your rubric, but one consistently produces formatted pages with headings and bullets while the other gives three plain sentences containing the same content. Which should you prefer, and why?","options":[{"text":"The formatted one, because structure makes answers easier to scan and reuse in documents.","correct":false,"tempting":"Values a house style set by a system prompt as if it were substance, when the same content is present in both."},{"text":"Neither on this basis; formatting is a matter of taste and the tie should be broken on price or data terms alone.","correct":false,"tempting":"Declares the difference irrelevant when the reading cost is real and falls on you every day the tool is used."},{"text":"The formatted one, because a model producing structured output is demonstrating better instruction-following on the same prompt.","correct":false,"tempting":"Reads unrequested structure as instruction-following, when the instruction came from the product rather than from you."},{"text":"The plain one, since the extra formatting is a system-prompt house style and length costs you reading time.","correct":true}]},"prev":{"slug":"same-prompt-twice","title":"Run it five times","url":"https://addaly.com/learn/choosing-your-tools/same-prompt-twice"},"next":{"slug":"reading-a-leaderboard","title":"Reading a leaderboard without being fooled","url":"https://addaly.com/learn/choosing-your-tools/reading-a-leaderboard"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A rubric you can apply in ten seconds', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/scoring-without-fooling-yourself (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A rubric you can apply in ten seconds","courseTitle":"Choosing and Using the Tools","module":{"index":2,"slug":"testing-it-yourself","title":"Testing a tool yourself"},"answer":null,"keyIdea":"Score each answer 0, 1 or 2 against criteria written before you saw any answers, and watch for the four biases (length, formatting, confidence and agreement) that inflate scores without improving usefulness.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-a-score-beats-an-impression","runs":[{"text":"Why a score beats an impression"}]},{"t":"p","runs":[{"text":"“That one felt better” is not portable. You cannot compare it to what you felt six months ago, you cannot explain it to a colleague, and you cannot tell whether it was the answer or your mood. A number, even a crude one, fixes all three."}]},{"t":"p","runs":[{"text":"The rubric that works for personal tool choice is deliberately blunt. Three points per answer:"}]},{"t":"ul","items":[[{"text":"2","b":true},{"text":": I would use this as it stands, or with a trivial edit."}],[{"text":"1","b":true},{"text":": Useful, but I would have to fix something before using it."}],[{"text":"0","b":true},{"text":": Wrong, or I would rewrite it from scratch."}]]},{"t":"p","runs":[{"text":"Twelve prompts, twenty-four points available, one number per tool. It takes ten seconds an answer and it is enough."}]},{"t":"h2","id":"the-pre-commitment-that-makes-it-work","runs":[{"text":"The pre-commitment that makes it work"}]},{"t":"p","runs":[{"text":"The rubric is not the hard part. The hard part is that you decided what “good” meant before you saw any answers, on the "},{"text":"good:","code":true},{"text":" line in your prompt file. Without that, scoring drifts: you read a fluent answer, it impresses you, and the criteria quietly rearrange themselves to match what you just read."}]},{"t":"p","runs":[{"text":"If you only take one habit from this module, take that one. Written criteria first, answers second."}]},{"t":"h2","id":"four-ways-scoring-goes-wrong","runs":[{"text":"Four ways scoring goes wrong"}]},{"t":"p","runs":[{"text":"Length.","b":true},{"text":" Longer answers score higher, from human and machine judges alike, largely regardless of merit. When two answers are both correct, the shorter is better; you have to read it."}]},{"t":"p","runs":[{"text":"Formatting.","b":true},{"text":" Headings, bold text and bullet points read as thorough. They are a house style set by a system prompt, not evidence of quality. A tool that returns three plain sentences containing everything you needed has beaten the one that returned a formatted page containing the same three sentences."}]},{"t":"p","runs":[{"text":"Confidence.","b":true},{"text":" A decisive wrong answer scores above a hedged right one, unless you are watching for it. Your known-answer probes are the correction, which is why they are in the file."}]},{"t":"p","runs":[{"text":"Agreement with you.","b":true},{"text":" An answer that confirms what you already thought feels correct. On anything where you have a prior view, force yourself to ask what you would say about this answer if it contradicted you."}]},{"t":"h2","id":"when-two-things-matter-separately","runs":[{"text":"When two things matter separately"}]},{"t":"p","runs":[{"text":"Sometimes one score hides the decision. If both correctness and tone matter to you and the tools split (one accurate and stiff, one warm and occasionally wrong) score two columns rather than averaging them into a number that describes neither."}]},{"t":"p","runs":[{"text":"Two columns is the maximum. Beyond that you are building an evaluation framework rather than choosing a tool, and the extra columns get filled in carelessly by the eighth prompt."}]},{"t":"h2","id":"scoring-answers-you-are-not-qualified-to-judge","runs":[{"text":"Scoring answers you are not qualified to judge"}]},{"t":"p","runs":[{"text":"There is one case where this rubric fails honestly, and pretending otherwise would be useless. If you cannot tell whether an answer is correct (a legal question outside your area, code in a language you do not read, a translation into a language you only half speak) then a score of 2 means “this looked good to me”, which is exactly the judgement these tools are best at defeating."}]},{"t":"p","runs":[{"text":"Two honest options. Either mark it "},{"text":"?","code":true},{"text":" and get one person who can judge it to look at three answers, or replace that prompt with one where you can verify the answer. Do not score it and pretend the number means something; a false 2 on the prompt you understand least will quietly dominate a close comparison."}]},{"t":"h2","id":"recording-the-result","runs":[{"text":"Recording the result"}]},{"t":"code","lang":"","label":"Text","text":"run 2026-03-14   12 prompts, blind, one sitting\n  toolA (gpt-4o-mini-2024-07-18)     19/24   fast, terse, weak on Hindi register\n  toolB (claude-sonnet-4-5)          21/24   best on the long contract\n  local (qwen3-8b, Q4_K_M, laptop)   14/24   fine on 1-6, lost on 7-9\ndecision: toolB paid, toolA free tier as second opinion, local for\n          anything with client names in it\nrevisit: 2026-09-14"},{"t":"p","runs":[{"text":"Six lines. That is a decision record, and it does three things an impression cannot: it tells your future self what you actually observed, it names the exact versions so you can tell later whether the tool changed or you did, and it states the decision so you stop relitigating it every time somebody posts a benchmark."}]},{"t":"h2","id":"the-honest-limit-of-all-this","runs":[{"text":"The honest limit of all this"}],"role":"limit"},{"t":"p","runs":[{"text":"Twelve prompts scored zero to two by one person is not a rigorous evaluation and does not pretend to be. It cannot estimate an error rate, it cannot detect a five per cent difference, and a different person with different work would reasonably reach a different answer."}]},{"t":"p","runs":[{"text":"It does not need to do any of that. It needs to be better than choosing on the basis of a headline, and it clears that bar by an enormous margin, at a cost of one hour twice a year."}]}],"check":{"question":[{"text":"Two tools tie on your rubric, but one consistently produces formatted pages with headings and bullets while the other gives three plain sentences containing the same content. Which should you prefer, and why?"}],"options":[{"text":[{"text":"The formatted one, because structure makes answers easier to scan and reuse in documents."}],"correct":false,"tempting":[{"text":"Values a house style set by a system prompt as if it were substance, when the same content is present in both."}]},{"text":[{"text":"Neither on this basis; formatting is a matter of taste and the tie should be broken on price or data terms alone."}],"correct":false,"tempting":[{"text":"Declares the difference irrelevant when the reading cost is real and falls on you every day the tool is used."}]},{"text":[{"text":"The formatted one, because a model producing structured output is demonstrating better instruction-following on the same prompt."}],"correct":false,"tempting":[{"text":"Reads unrequested structure as instruction-following, when the instruction came from the product rather than from you."}]},{"text":[{"text":"The plain one, since the extra formatting is a system-prompt house style and length costs you reading time."}],"correct":true,"tempting":null}],"order":[2,1,3,0],"why":null},"prev":{"slug":"same-prompt-twice","title":"Run it five times","minutes":8},"next":{"slug":"reading-a-leaderboard","title":"Reading a leaderboard without being fooled","minutes":9}}}