When your data is too small, and what to do instead

The arithmetic of small numbers

You run a promotion. Forty customers before, forty after. Conversion rises from 25% to 35%.

That looks like a real improvement, and with these numbers it is entirely consistent with nothing having happened. With samples of forty, a ten-point swing arrives by chance often enough that you should expect to see one regularly without any cause at all.

This is not a reason to stop looking at your data. It is a reason to hold conclusions loosely at these volumes, and to notice that the feeling of certainty produced by a chart is not proportional to the evidence in it.

A rough rule for percentages: the uncertainty is about 100 divided by the square root of your sample, in points. Forty customers gives roughly plus or minus 16 points. Four hundred gives about 5. A thousand gives about 3. That single line of arithmetic will save you from most of what follows.

The mistake nobody thinks of as a mistake

You look at your data. You check whether Tuesdays are worse. They are not. You check whether the new window display helped. Unclear. You check whether customers from the north side spend more. Not really. You check eleven more things. The twelfth shows a striking difference, and that is the one you remember and act on.

Testing twenty independent ideas at the usual standard of evidence gives you roughly a one-in-three chance of at least one false finding. Testing forty makes a false finding more likely than not.

Nobody does this dishonestly. It is what curiosity looks like when applied to one dataset, and it is why the pattern you found by rummaging deserves much less confidence than the one you predicted in advance.

The defence: predict first

Write down what you expect before you look. “I think Saturday afternoons are our weakest slot.” Then check.

A confirmed prediction is real evidence. A pattern found by scanning is a hypothesis: interesting, worth testing on new data, not worth restocking the shop for.

The practical version is to keep a short note of what you expected and what you found. Over a year it tells you something valuable and slightly humbling about how well you know your own business.

The alternative that actually works: change something forward

When the data cannot answer the question, stop asking it and run the thing instead.

Decide the change, decide in advance how long and what you will measure, run it, and look once at the end. Two weeks minimum so you cover both halves of the week-of-month effect; four if your trade is weather-dependent.

The trap is stopping early because it is going well. Peeking and stopping when the numbers look good turns a coin-flipping exercise into a positive result nearly every time, and it is the most common way small-business experiments produce confident nonsense. Decide the end date, and honour it.

Formal A/B testing, and why it mostly does not apply

Detecting a genuine improvement from 3% to 4% conversion at conventional standards needs several thousand people per side. A small business does not have them, and will not have them this quarter.

What this means practically is not that you cannot learn anything. It means you should only expect to detect large effects. A change that doubles enquiries will show up in your numbers within a fortnight. A change worth 1% will not be detectable this year by any method, which is a reason to stop trying to measure it, not a reason to avoid making it.

So spend your measurement effort on big swings, and make the small improvements because they are obviously sensible, not because you intend to prove them.

What to do instead of modelling, at small volumes

Count things and look at them. Averages hide everything interesting; ranges and distributions do not.

Read the raw material. Fifty customer enquiries, read properly, will teach you more than a model built on five hundred rows, because the enquiries contain the reasons and the rows contain only the outcomes.

Ask people. The three customers who stopped coming know exactly why, and none of it is in your till data.

The honest conclusion of this block

For a large number of small businesses, the right answer to “should we build a model?” is no, and the right answer to “should we look properly at our own numbers?” is yes, urgently, and it will take an afternoon.

That is not a disappointing outcome. Most of the value in this module is in the first three lessons (knowing what you have, cleaning it, and computing honestly) and those pay off at any size.