{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"blind-comparison","title":"Taking the brand off the answers","url":"https://addaly.com/learn/choosing-your-tools/blind-comparison","position":14,"of":76,"minutes":7,"answer":"Score answers labelled A, B and C against criteria written in advance and only then reveal which tool produced which, because brand halo, order and length shift judgements more than the tools differ.","keyIdea":"Score answers labelled A, B and C against criteria written in advance and only then reveal which tool produced which, because brand halo, order and length shift judgements more than the tools differ."},"body":{"format":"markdown","text":"## The halo is bigger than the gap\n\nAsk ten people to compare two answers with the logos visible, and the brand they already trust wins more often than the answers justify. This is not a weakness peculiar to you. It is one of the most reliably reproduced findings in the study of judgement, and it applies with particular force here, because everyone has already absorbed opinions about these companies from months of headlines.\n\nThe fix costs five minutes and it will change at least one of your conclusions.\n\n## The method\n\n1. Run each of your twelve prompts through each candidate tool. Do it in one sitting.\n2. Paste the answers into one document, labelled only `A`, `B`, `C`, with the mapping written on a separate piece of paper you turn face down.\n3. Shuffle which letter goes first for each prompt, so `A` is not always at the top.\n4. Score every answer against the `good:` line you wrote earlier, before you look at the mapping.\n5. Only then turn the paper over.\n\nThat is the whole thing. It is the same method a wine tasting uses, for the same reason.\n\n## Three biases it removes, and one it does not\n\n**Brand halo.** The one described above.\n\n**Order effects.** Whichever answer you read first sets your expectation, and the second is judged against it. Shuffling handles this.\n\n**Length bias.** Longer answers are rated higher, by human judges and by AI judges alike, largely independent of whether they are better. This one is worth explicit resistance: when two answers are equally correct and one is twice as long, the shorter one is better for almost every real purpose, because you have to read it.\n\nThe bias blinding does not remove is **style preference**. If one model writes in a rhythm you enjoy, you will prefer its answers even blind. That is not a flaw in the test: you have to read this output every day and enjoying it has value. Just be honest about naming it as taste rather than quality when you record the result.\n\n## Doing it properly on a phone\n\nMost people testing tools are doing so on a phone, and there is no reason this needs a computer. Open a note. Paste each answer under a heading `A`, `B`, `C`. Write your scores in the same note. The mapping goes in a second note you do not open until the end. The discipline is the same; the equipment does not matter.\n\n## Getting the answers in the first place\n\nOne practical trap. If you run the same prompt in three tools while logged into accounts that carry memory, custom instructions and past conversations, you are not comparing the tools: you are comparing three differently configured versions of your own history. A tool that has been told for six months that you work in logistics will look better on a logistics prompt.\n\nFor a fair run, use a fresh or private window for each, turn off memory and custom instructions where you can, and paste the same context into all three. It costs a few minutes and it removes the largest remaining confound in a home-made comparison.\n\n## The comparison that is not worth doing\n\nThere is a version of this that wastes an afternoon: running fifty prompts across six tools. Beyond three candidates and a dozen prompts, you are collecting data you will not act on, and the fatigue in the last hour makes the later scores worse than the earlier ones.\n\nThree tools, twelve prompts, one sitting, one hour. If two of them tie, you have learned something real: for your work, the choice does not matter, and you should pick on price, on data terms, or on which one is easier to reach from your country.\n\n## What \"wins\" means\n\nAdd up the scores and you will usually find no landslide. A typical honest result is 21, 19 and 14 out of 24: two tools that are effectively equivalent for you and one that is not.\n\nThat result is worth more than a ranking, because it tells you which comparisons to stop having. The two near-equals are interchangeable, so choose between them on cost and terms and stop reading arguments about them. And the gap to the third is real enough to act on.\n\nWrite the numbers, the date and the exact model strings into your file. In six months you will run it again, and having last time's numbers in front of you is the difference between measuring a change and remembering one.\n\n> Figure: A typical honest result: twelve prompts, scored blind. Bars (points out of 24): Tool B, paid plan 21; Tool A, free tier 19; Tool C 14; An 8B model on the laptop 12. No landslide. The top two are effectively interchangeable for this person's work, so the choice between them is price, data terms and whether the card works from here, and the argument about which is better can stop. The gap down to the third is real enough to act on."},"check":{"question":"In a blind run, two tools score 21 and 19 out of 24 and a third scores 14. What is the most useful conclusion?","options":[{"text":"The 21 is the best tool and should be adopted, since the ranking is unambiguous across twelve varied prompts.","correct":false,"tempting":"Reads a two-point gap on twelve items as a real ordering, when that margin is inside what scoring noise on a sample this small produces."},{"text":"The test needs more prompts, because twelve items cannot separate 21 from 19 with any confidence.","correct":false,"tempting":"Chases a distinction that would not change the decision; more items would sharpen a gap too small to act on either way."},{"text":"The top two are interchangeable here, so decide on price or terms; the gap to the third is real.","correct":true},{"text":"The third tool should be re-tested unblinded to check whether the low score came from an unfamiliar writing style.","correct":false,"tempting":"Reintroduces exactly the bias the blinding removed, and a seven-point deficit is far beyond what style preference explains."}]},"prev":{"slug":"testing-your-language","title":"Testing the language you actually work in","url":"https://addaly.com/learn/choosing-your-tools/testing-your-language"},"next":{"slug":"same-prompt-twice","title":"Run it five times","url":"https://addaly.com/learn/choosing-your-tools/same-prompt-twice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Taking the brand off the answers', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/blind-comparison (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Taking the brand off the answers","courseTitle":"Choosing and Using the Tools","module":{"index":2,"slug":"testing-it-yourself","title":"Testing a tool yourself"},"answer":null,"keyIdea":"Score answers labelled A, B and C against criteria written in advance and only then reveal which tool produced which, because brand halo, order and length shift judgements more than the tools differ.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"points out of 24","title":"A typical honest result: twelve prompts, scored blind","series":[{"label":"Tool B, paid plan","value":21,"highlight":true},{"label":"Tool A, free tier","value":19},{"label":"Tool C","value":14},{"label":"An 8B model on the laptop","value":12}],"caption":"No landslide. The top two are effectively interchangeable for this person’s work, so the choice between them is price, data terms and whether the card works from here, and the argument about which is better can stop. The gap down to the third is real enough to act on."},"alt":"Figure: A typical honest result: twelve prompts, scored blind. Bars (points out of 24): Tool B, paid plan 21; Tool A, free tier 19; Tool C 14; An 8B model on the laptop 12. No landslide. The top two are effectively interchangeable for this person’s work, so the choice between them is price, data terms and whether the card works from here, and the argument about which is better can stop. The gap down to the third is real enough to act on."}},"blocks":[{"t":"h2","id":"the-halo-is-bigger-than-the-gap","runs":[{"text":"The halo is bigger than the gap"}]},{"t":"p","runs":[{"text":"Ask ten people to compare two answers with the logos visible, and the brand they already trust wins more often than the answers justify. This is not a weakness peculiar to you. It is one of the most reliably reproduced findings in the study of judgement, and it applies with particular force here, because everyone has already absorbed opinions about these companies from months of headlines."}]},{"t":"p","runs":[{"text":"The fix costs five minutes and it will change at least one of your conclusions."}]},{"t":"h2","id":"the-method","runs":[{"text":"The method"}]},{"t":"ol","start":1,"items":[[{"text":"Run each of your twelve prompts through each candidate tool. Do it in one sitting."}],[{"text":"Paste the answers into one document, labelled only "},{"text":"A","code":true},{"text":", "},{"text":"B","code":true},{"text":", "},{"text":"C","code":true},{"text":", with the mapping written on a separate piece of paper you turn face down."}],[{"text":"Shuffle which letter goes first for each prompt, so "},{"text":"A","code":true},{"text":" is not always at the top."}],[{"text":"Score every answer against the "},{"text":"good:","code":true},{"text":" line you wrote earlier, before you look at the mapping."}],[{"text":"Only then turn the paper over."}]]},{"t":"p","runs":[{"text":"That is the whole thing. It is the same method a wine tasting uses, for the same reason."}]},{"t":"h2","id":"three-biases-it-removes-and-one-it-does-not","runs":[{"text":"Three biases it removes, and one it does not"}]},{"t":"p","runs":[{"text":"Brand halo.","b":true},{"text":" The one described above."}]},{"t":"p","runs":[{"text":"Order effects.","b":true},{"text":" Whichever answer you read first sets your expectation, and the second is judged against it. Shuffling handles this."}]},{"t":"p","runs":[{"text":"Length bias.","b":true},{"text":" Longer answers are rated higher, by human judges and by AI judges alike, largely independent of whether they are better. This one is worth explicit resistance: when two answers are equally correct and one is twice as long, the shorter one is better for almost every real purpose, because you have to read it."}]},{"t":"p","runs":[{"text":"The bias blinding does not remove is "},{"text":"style preference","b":true},{"text":". If one model writes in a rhythm you enjoy, you will prefer its answers even blind. That is not a flaw in the test: you have to read this output every day and enjoying it has value. Just be honest about naming it as taste rather than quality when you record the result."}]},{"t":"h2","id":"doing-it-properly-on-a-phone","runs":[{"text":"Doing it properly on a phone"}]},{"t":"p","runs":[{"text":"Most people testing tools are doing so on a phone, and there is no reason this needs a computer. Open a note. Paste each answer under a heading "},{"text":"A","code":true},{"text":", "},{"text":"B","code":true},{"text":", "},{"text":"C","code":true},{"text":". Write your scores in the same note. The mapping goes in a second note you do not open until the end. The discipline is the same; the equipment does not matter."}]},{"t":"h2","id":"getting-the-answers-in-the-first-place","runs":[{"text":"Getting the answers in the first place"}]},{"t":"p","runs":[{"text":"One practical trap. If you run the same prompt in three tools while logged into accounts that carry memory, custom instructions and past conversations, you are not comparing the tools: you are comparing three differently configured versions of your own history. A tool that has been told for six months that you work in logistics will look better on a logistics prompt."}]},{"t":"p","runs":[{"text":"For a fair run, use a fresh or private window for each, turn off memory and custom instructions where you can, and paste the same context into all three. It costs a few minutes and it removes the largest remaining confound in a home-made comparison."}]},{"t":"h2","id":"the-comparison-that-is-not-worth-doing","runs":[{"text":"The comparison that is not worth doing"}]},{"t":"p","runs":[{"text":"There is a version of this that wastes an afternoon: running fifty prompts across six tools. Beyond three candidates and a dozen prompts, you are collecting data you will not act on, and the fatigue in the last hour makes the later scores worse than the earlier ones."}]},{"t":"p","runs":[{"text":"Three tools, twelve prompts, one sitting, one hour. If two of them tie, you have learned something real: for your work, the choice does not matter, and you should pick on price, on data terms, or on which one is easier to reach from your country."}]},{"t":"h2","id":"what-wins-means","runs":[{"text":"What “wins” means"}]},{"t":"p","runs":[{"text":"Add up the scores and you will usually find no landslide. A typical honest result is 21, 19 and 14 out of 24: two tools that are effectively equivalent for you and one that is not."}]},{"t":"p","runs":[{"text":"That result is worth more than a ranking, because it tells you which comparisons to stop having. The two near-equals are interchangeable, so choose between them on cost and terms and stop reading arguments about them. And the gap to the third is real enough to act on."}]},{"t":"p","runs":[{"text":"Write the numbers, the date and the exact model strings into your file. In six months you will run it again, and having last time’s numbers in front of you is the difference between measuring a change and remembering one."}]}],"check":{"question":[{"text":"In a blind run, two tools score 21 and 19 out of 24 and a third scores 14. What is the most useful conclusion?"}],"options":[{"text":[{"text":"The 21 is the best tool and should be adopted, since the ranking is unambiguous across twelve varied prompts."}],"correct":false,"tempting":[{"text":"Reads a two-point gap on twelve items as a real ordering, when that margin is inside what scoring noise on a sample this small produces."}]},{"text":[{"text":"The test needs more prompts, because twelve items cannot separate 21 from 19 with any confidence."}],"correct":false,"tempting":[{"text":"Chases a distinction that would not change the decision; more items would sharpen a gap too small to act on either way."}]},{"text":[{"text":"The top two are interchangeable here, so decide on price or terms; the gap to the third is real."}],"correct":true,"tempting":null},{"text":[{"text":"The third tool should be re-tested unblinded to check whether the low score came from an unfamiliar writing style."}],"correct":false,"tempting":[{"text":"Reintroduces exactly the bias the blinding removed, and a seven-point deficit is far beyond what style preference explains."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"testing-your-language","title":"Testing the language you actually work in","minutes":9},"next":{"slug":"same-prompt-twice","title":"Run it five times","minutes":8}}}