Taking the brand off the answers

The halo is bigger than the gap

Ask ten people to compare two answers with the logos visible, and the brand they already trust wins more often than the answers justify. This is not a weakness peculiar to you. It is one of the most reliably reproduced findings in the study of judgement, and it applies with particular force here, because everyone has already absorbed opinions about these companies from months of headlines.

The fix costs five minutes and it will change at least one of your conclusions.

The method

1. Run each of your twelve prompts through each candidate tool. Do it in one sitting.
2. Paste the answers into one document, labelled only A, B, C, with the mapping written on a separate piece of paper you turn face down.
3. Shuffle which letter goes first for each prompt, so A is not always at the top.
4. Score every answer against the good: line you wrote earlier, before you look at the mapping.
5. Only then turn the paper over.

That is the whole thing. It is the same method a wine tasting uses, for the same reason.

Three biases it removes, and one it does not

Brand halo. The one described above.

Order effects. Whichever answer you read first sets your expectation, and the second is judged against it. Shuffling handles this.

Length bias. Longer answers are rated higher, by human judges and by AI judges alike, largely independent of whether they are better. This one is worth explicit resistance: when two answers are equally correct and one is twice as long, the shorter one is better for almost every real purpose, because you have to read it.

The bias blinding does not remove is style preference. If one model writes in a rhythm you enjoy, you will prefer its answers even blind. That is not a flaw in the test: you have to read this output every day and enjoying it has value. Just be honest about naming it as taste rather than quality when you record the result.

Doing it properly on a phone

Most people testing tools are doing so on a phone, and there is no reason this needs a computer. Open a note. Paste each answer under a heading A, B, C. Write your scores in the same note. The mapping goes in a second note you do not open until the end. The discipline is the same; the equipment does not matter.

Getting the answers in the first place

One practical trap. If you run the same prompt in three tools while logged into accounts that carry memory, custom instructions and past conversations, you are not comparing the tools: you are comparing three differently configured versions of your own history. A tool that has been told for six months that you work in logistics will look better on a logistics prompt.

For a fair run, use a fresh or private window for each, turn off memory and custom instructions where you can, and paste the same context into all three. It costs a few minutes and it removes the largest remaining confound in a home-made comparison.

The comparison that is not worth doing

There is a version of this that wastes an afternoon: running fifty prompts across six tools. Beyond three candidates and a dozen prompts, you are collecting data you will not act on, and the fatigue in the last hour makes the later scores worse than the earlier ones.

Three tools, twelve prompts, one sitting, one hour. If two of them tie, you have learned something real: for your work, the choice does not matter, and you should pick on price, on data terms, or on which one is easier to reach from your country.

What “wins” means

Add up the scores and you will usually find no landslide. A typical honest result is 21, 19 and 14 out of 24: two tools that are effectively equivalent for you and one that is not.

That result is worth more than a ranking, because it tells you which comparisons to stop having. The two near-equals are interchangeable, so choose between them on cost and terms and stop reading arguments about them. And the gap to the third is real enough to act on.

Write the numbers, the date and the exact model strings into your file. In six months you will run it again, and having last time’s numbers in front of you is the difference between measuring a change and remembering one.