{"course":{"slug":"ai-for-small-business","title":"AI for a Small Business","url":"https://addaly.com/learn/ai-for-small-business","level":"beginner","track":"work"},"lesson":{"slug":"small-numbers-and-superstition","title":"When your data is too small, and what to do instead","url":"https://addaly.com/learn/ai-for-small-business/small-numbers-and-superstition","position":76,"of":88,"minutes":9,"answer":"Small samples produce large swings by chance and a busy owner tests many ideas against the same data, so most patterns found this way are noise.","keyIdea":"Small samples produce large swings by chance and a busy owner tests many ideas against the same data, so most patterns found this way are noise, which is why a deliberate change measured forwards beats a pattern discovered backwards."},"body":{"format":"markdown","text":"## The arithmetic of small numbers\n\nYou run a promotion. Forty customers before, forty after. Conversion rises from 25% to 35%.\n\nThat looks like a real improvement, and with these numbers it is entirely consistent with nothing having happened. With samples of forty, a ten-point swing arrives by chance often enough that you should expect to see one regularly without any cause at all.\n\nThis is not a reason to stop looking at your data. It is a reason to hold conclusions loosely at these volumes, and to notice that the feeling of certainty produced by a chart is not proportional to the evidence in it.\n\nA rough rule for percentages: the uncertainty is about 100 divided by the square root of your sample, in points. Forty customers gives roughly plus or minus 16 points. Four hundred gives about 5. A thousand gives about 3. That single line of arithmetic will save you from most of what follows.\n\n## The mistake nobody thinks of as a mistake\n\nYou look at your data. You check whether Tuesdays are worse. They are not. You check whether the new window display helped. Unclear. You check whether customers from the north side spend more. Not really. You check eleven more things. The twelfth shows a striking difference, and that is the one you remember and act on.\n\nTesting twenty independent ideas at the usual standard of evidence gives you roughly a one-in-three chance of at least one false finding. Testing forty makes a false finding more likely than not.\n\nNobody does this dishonestly. It is what curiosity looks like when applied to one dataset, and it is why the pattern you found by rummaging deserves much less confidence than the one you predicted in advance.\n\n## The defence: predict first\n\nWrite down what you expect before you look. \"I think Saturday afternoons are our weakest slot.\" Then check.\n\nA confirmed prediction is real evidence. A pattern found by scanning is a hypothesis: interesting, worth testing on new data, not worth restocking the shop for.\n\nThe practical version is to keep a short note of what you expected and what you found. Over a year it tells you something valuable and slightly humbling about how well you know your own business.\n\n## The alternative that actually works: change something forward\n\nWhen the data cannot answer the question, stop asking it and run the thing instead.\n\nDecide the change, decide in advance how long and what you will measure, run it, and look once at the end. Two weeks minimum so you cover both halves of the week-of-month effect; four if your trade is weather-dependent.\n\nThe trap is stopping early because it is going well. Peeking and stopping when the numbers look good turns a coin-flipping exercise into a positive result nearly every time, and it is the most common way small-business experiments produce confident nonsense. Decide the end date, and honour it.\n\n## Formal A/B testing, and why it mostly does not apply\n\nDetecting a genuine improvement from 3% to 4% conversion at conventional standards needs several thousand people per side. A small business does not have them, and will not have them this quarter.\n\nWhat this means practically is not that you cannot learn anything. It means you should only expect to detect large effects. A change that doubles enquiries will show up in your numbers within a fortnight. A change worth 1% will not be detectable this year by any method, which is a reason to stop trying to measure it, not a reason to avoid making it.\n\nSo spend your measurement effort on big swings, and make the small improvements because they are obviously sensible, not because you intend to prove them.\n\n## What to do instead of modelling, at small volumes\n\nCount things and look at them. Averages hide everything interesting; ranges and distributions do not.\n\nRead the raw material. Fifty customer enquiries, read properly, will teach you more than a model built on five hundred rows, because the enquiries contain the reasons and the rows contain only the outcomes.\n\nAsk people. The three customers who stopped coming know exactly why, and none of it is in your till data.\n\n## The honest conclusion of this block\n\nFor a large number of small businesses, the right answer to \"should we build a model?\" is no, and the right answer to \"should we look properly at our own numbers?\" is yes, urgently, and it will take an afternoon.\n\nThat is not a disappointing outcome. Most of the value in this module is in the first three lessons (knowing what you have, cleaning it, and computing honestly) and those pay off at any size."},"check":{"question":"A shop tries a new window display and compares the two weeks after with the two weeks before: 52 sales against 44. The owner concludes it works. What is the strongest reason to hold off?","options":[{"text":"Two weeks is too short to cover the week-of-month effect, so the comparison may be measuring pay dates rather than the display.","correct":false,"tempting":"A genuine confound, and two weeks each side does span the cycle; the size of the sample is the larger problem."},{"text":"The comparison lacks a control, since trade may have risen across the whole high street for reasons unrelated to the display.","correct":false,"tempting":"A real weakness of before-and-after designs, though a small shop rarely has a usable control and the numbers are too small either way."},{"text":"Sales are the wrong measure, because a window display acts on footfall and an unchanged conversion rate would mean the display did nothing.","correct":false,"tempting":"A sharper measure would help, and it does not rescue a difference this size from chance."},{"text":"A difference of eight on totals near fifty is well within what chance produces, so the figures are consistent with no effect.","correct":true}]},"prev":{"slug":"automl-and-what-it-hides","title":"AutoML: what it genuinely does, and what it cannot do for you","url":"https://addaly.com/learn/ai-for-small-business/automl-and-what-it-hides"},"next":{"slug":"what-it-costs","title":"What It Costs, and Keeping It Near Zero","url":"https://addaly.com/learn/ai-for-small-business/what-it-costs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When your data is too small, and what to do instead', in AI for a Small Business. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-for-small-business/small-numbers-and-superstition (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When your data is too small, and what to do instead","courseTitle":"AI for a Small Business","module":{"index":8,"slug":"your-own-numbers","title":"Learning from your own numbers"},"answer":null,"keyIdea":"Small samples produce large swings by chance and a busy owner tests many ideas against the same data, so most patterns found this way are noise, which is why a deliberate change measured forwards beats a pattern discovered backwards.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-arithmetic-of-small-numbers","runs":[{"text":"The arithmetic of small numbers"}]},{"t":"p","runs":[{"text":"You run a promotion. Forty customers before, forty after. Conversion rises from 25% to 35%."}]},{"t":"p","runs":[{"text":"That looks like a real improvement, and with these numbers it is entirely consistent with nothing having happened. With samples of forty, a ten-point swing arrives by chance often enough that you should expect to see one regularly without any cause at all."}]},{"t":"p","runs":[{"text":"This is not a reason to stop looking at your data. It is a reason to hold conclusions loosely at these volumes, and to notice that the feeling of certainty produced by a chart is not proportional to the evidence in it."}]},{"t":"p","runs":[{"text":"A rough rule for percentages: the uncertainty is about 100 divided by the square root of your sample, in points. Forty customers gives roughly plus or minus 16 points. Four hundred gives about 5. A thousand gives about 3. That single line of arithmetic will save you from most of what follows."}]},{"t":"h2","id":"the-mistake-nobody-thinks-of-as-a-mistake","runs":[{"text":"The mistake nobody thinks of as a mistake"}]},{"t":"p","runs":[{"text":"You look at your data. You check whether Tuesdays are worse. They are not. You check whether the new window display helped. Unclear. You check whether customers from the north side spend more. Not really. You check eleven more things. The twelfth shows a striking difference, and that is the one you remember and act on."}]},{"t":"p","runs":[{"text":"Testing twenty independent ideas at the usual standard of evidence gives you roughly a one-in-three chance of at least one false finding. Testing forty makes a false finding more likely than not."}]},{"t":"p","runs":[{"text":"Nobody does this dishonestly. It is what curiosity looks like when applied to one dataset, and it is why the pattern you found by rummaging deserves much less confidence than the one you predicted in advance."}]},{"t":"h2","id":"the-defence-predict-first","runs":[{"text":"The defence: predict first"}]},{"t":"p","runs":[{"text":"Write down what you expect before you look. “I think Saturday afternoons are our weakest slot.” Then check."}]},{"t":"p","runs":[{"text":"A confirmed prediction is real evidence. A pattern found by scanning is a hypothesis: interesting, worth testing on new data, not worth restocking the shop for."}]},{"t":"p","runs":[{"text":"The practical version is to keep a short note of what you expected and what you found. Over a year it tells you something valuable and slightly humbling about how well you know your own business."}]},{"t":"h2","id":"the-alternative-that-actually-works-change-something-forward","runs":[{"text":"The alternative that actually works: change something forward"}]},{"t":"p","runs":[{"text":"When the data cannot answer the question, stop asking it and run the thing instead."}]},{"t":"p","runs":[{"text":"Decide the change, decide in advance how long and what you will measure, run it, and look once at the end. Two weeks minimum so you cover both halves of the week-of-month effect; four if your trade is weather-dependent."}]},{"t":"p","runs":[{"text":"The trap is stopping early because it is going well. Peeking and stopping when the numbers look good turns a coin-flipping exercise into a positive result nearly every time, and it is the most common way small-business experiments produce confident nonsense. Decide the end date, and honour it."}]},{"t":"h2","id":"formal-a-b-testing-and-why-it-mostly-does-not-apply","runs":[{"text":"Formal A/B testing, and why it mostly does not apply"}]},{"t":"p","runs":[{"text":"Detecting a genuine improvement from 3% to 4% conversion at conventional standards needs several thousand people per side. A small business does not have them, and will not have them this quarter."}]},{"t":"p","runs":[{"text":"What this means practically is not that you cannot learn anything. It means you should only expect to detect large effects. A change that doubles enquiries will show up in your numbers within a fortnight. A change worth 1% will not be detectable this year by any method, which is a reason to stop trying to measure it, not a reason to avoid making it."}]},{"t":"p","runs":[{"text":"So spend your measurement effort on big swings, and make the small improvements because they are obviously sensible, not because you intend to prove them."}]},{"t":"h2","id":"what-to-do-instead-of-modelling-at-small-volumes","runs":[{"text":"What to do instead of modelling, at small volumes"}]},{"t":"p","runs":[{"text":"Count things and look at them. Averages hide everything interesting; ranges and distributions do not."}]},{"t":"p","runs":[{"text":"Read the raw material. Fifty customer enquiries, read properly, will teach you more than a model built on five hundred rows, because the enquiries contain the reasons and the rows contain only the outcomes."}]},{"t":"p","runs":[{"text":"Ask people. The three customers who stopped coming know exactly why, and none of it is in your till data."}]},{"t":"h2","id":"the-honest-conclusion-of-this-block","runs":[{"text":"The honest conclusion of this block"}]},{"t":"p","runs":[{"text":"For a large number of small businesses, the right answer to “should we build a model?” is no, and the right answer to “should we look properly at our own numbers?” is yes, urgently, and it will take an afternoon."}]},{"t":"p","runs":[{"text":"That is not a disappointing outcome. Most of the value in this module is in the first three lessons (knowing what you have, cleaning it, and computing honestly) and those pay off at any size."}]}],"check":{"question":[{"text":"A shop tries a new window display and compares the two weeks after with the two weeks before: 52 sales against 44. The owner concludes it works. What is the strongest reason to hold off?"}],"options":[{"text":[{"text":"Two weeks is too short to cover the week-of-month effect, so the comparison may be measuring pay dates rather than the display."}],"correct":false,"tempting":[{"text":"A genuine confound, and two weeks each side does span the cycle; the size of the sample is the larger problem."}]},{"text":[{"text":"The comparison lacks a control, since trade may have risen across the whole high street for reasons unrelated to the display."}],"correct":false,"tempting":[{"text":"A real weakness of before-and-after designs, though a small shop rarely has a usable control and the numbers are too small either way."}]},{"text":[{"text":"Sales are the wrong measure, because a window display acts on footfall and an unchanged conversion rate would mean the display did nothing."}],"correct":false,"tempting":[{"text":"A sharper measure would help, and it does not rescue a difference this size from chance."}]},{"text":[{"text":"A difference of eight on totals near fifty is well within what chance produces, so the figures are consistent with no effect."}],"correct":true,"tempting":null}],"order":[2,1,3,0],"why":null},"prev":{"slug":"automl-and-what-it-hides","title":"AutoML: what it genuinely does, and what it cannot do for you","minutes":9},"next":{"slug":"what-it-costs","title":"What It Costs, and Keeping It Near Zero","minutes":8}}}