{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"clustered-and-repeated-data","title":"Five hundred rows, forty users","url":"https://addaly.com/learn/evaluating-ai/clustered-and-repeated-data","position":25,"of":76,"minutes":9,"answer":"When rows share a source (the same user, the same document, repeated runs of the same item) your effective sample size is closer to the number of sources than the number of rows.","keyIdea":"When rows share a source (the same user, the same document, repeated runs of the same item) your effective sample size is closer to the number of sources than the number of rows, so aggregate to the source before you compute anything."},"body":{"format":"markdown","text":"## The interval that keeps being wrong\n\nA team measures a support assistant on 500 logged conversations. The pass rate is 84%, the reported interval is plus or minus 3 points, and yet each fresh sample lands 6 or 7 points away. Nothing is broken in the arithmetic. The 500 conversations came from 40 customers, and rows from one customer resemble each other.\n\nEvery standard error formula assumes independent observations. Correlated rows carry less information than independent ones, and the amount less has a name.\n\n## The design effect, in numbers\n\nWith `m` rows per cluster and an intra-cluster correlation `ICC`, the variance is inflated by:\n\n```\ndesign effect = 1 + (m - 1) * ICC\n```\n\nFor 500 rows from 40 customers, `m` is 12.5. Suppose the ICC is 0.3, which is unremarkable for text from the same organisation:\n\n```\ndesign effect = 1 + 11.5 * 0.3 = 4.45\neffective n   = 500 / 4.45     ≈ 112\n```\n\nYou have 500 rows and about 112 rows' worth of information. The honest interval is `sqrt(4.45)`, a bit over twice, wider than the one printed, which is precisely the discrepancy the team kept seeing.\n\nThe same arithmetic covers the other common case: 100 items run 5 times each at temperature 0.7. Those 500 rows are not 500 independent observations either, and the clusters are the items.\n\n## Three fixes, in order of preference\n\n**Aggregate to the cluster.** Compute the rate per customer, then average across customers, and do all the statistics on those 40 numbers. It is simple, it is safe, and every formula becomes valid again because the 40 values are independent. The cost is a wider, correct interval.\n\n```python\nper_customer = df.groupby(\"customer_id\")[\"passed\"].mean()\nn = len(per_customer)                               # 40, not 500\nse = per_customer.std(ddof=1) / np.sqrt(n)\n```\n\n**Cluster bootstrap.** Resample customers with replacement, carrying all of each customer's rows, and recompute. Use this when the statistic is not a simple mean: recall at 5, a weighted composite, a per-item pass rate you want to weight by volume.\n\n**Cluster-robust standard errors or a mixed model.** `statsmodels` will fit both. Worth it when you need to adjust for covariates at the same time; unnecessary otherwise.\n\n## Finding the clusters, which are rarely labelled\n\nThe customer column is the easy case. The clusters that cause trouble are the ones nobody wrote down:\n\n- **One document, many questions.** Forty questions generated from the same PDF share its formatting, its vocabulary and its ambiguities. The unit is the document.\n- **One template, many variants.** Everything the previous module said about synthetic items applies here: forty rephrasings of one request are one observation wearing forty coats.\n- **One session, many turns.** A conversation that goes wrong at turn two produces five failed turns, which is one failure.\n- **One time window.** Everything logged during an outage shares the outage.\n\nBefore computing an interval, ask what the rows have in common that the population would not. If the answer is \"quite a lot\", find the grouping key and aggregate to it.\n\n## Repeated runs deserve their own decomposition\n\nWhen each item is run several times, two different variances are in play and they suggest different actions:\n\n- **Item-to-item variance**: some items are harder. More runs will not reduce it; more *items* will.\n- **Run-to-run variance**: the same item passing on Tuesday and failing on Wednesday. More items will not reduce it; more runs, a lower temperature, or a fixed seed will.\n\nCompute both. Take the per-item mean across runs, and look at the spread of those means (item variance) alongside the average spread within an item (run variance). If run-to-run spread is 5 points, then no amount of extra data makes a 3-point improvement detectable, and the correct next step is to make the system more deterministic rather than to label more.\n\n## Say both numbers out loud\n\nThe habit that prevents all of this is one line in every report: state the number of rows and the number of independent units.\n\n> Pass rate 84% across 500 conversations from 40 customers (95% CI 76 to 92, computed at the customer level).\n\nA reader who sees \"40 customers\" immediately knows the resolution available. A reader who sees only \"500 conversations\" will believe the study is four times more precise than it is.\n\n## The limit you cannot compute your way out of\n\nIf the set contains 8 customers, no statistical technique makes it behave like 8,000. Clustering is a data-collection problem with a statistical symptom. When the effective sample is small, the truthful report says so and recommends widening collection (more customers, more documents, more languages) rather than quoting a narrow interval that the next sample will contradict."},"check":{"question":"An evaluation shows 84% on 500 conversations drawn from 40 customers. Which reported figure is most defensible?","options":[{"text":"84% with a 3-point interval, since 500 items is a large sample by evaluation standards.","correct":false,"tempting":"Counts rows as independent observations; conversations from one customer are correlated, so 500 rows carry far less information than 500 independent draws."},{"text":"84% with no interval, since clustered data cannot support a confidence interval.","correct":false,"tempting":"Abandons the estimate rather than computing it at the right level; the customer-level mean supports a perfectly ordinary interval."},{"text":"84% averaged across customers, with the interval computed from the spread of the 40 customer-level rates.","correct":true},{"text":"84% with the interval computed after removing the customers contributing the most conversations, so no single customer dominates.","correct":false,"tempting":"Discarding data to reduce imbalance loses information and still treats the remaining rows as independent, which is the actual source of the error."}]},"prev":{"slug":"bootstrap-pitfalls","title":"The bootstrap, and the three places it lies","url":"https://addaly.com/learn/evaluating-ai/bootstrap-pitfalls"},"next":{"slug":"many-comparisons","title":"Twenty slices and one guaranteed surprise","url":"https://addaly.com/learn/evaluating-ai/many-comparisons"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Five hundred rows, forty users', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/clustered-and-repeated-data (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Five hundred rows, forty users","courseTitle":"Knowing If It Works","module":{"index":3,"slug":"deciding-if-it-is-real","title":"Deciding whether a difference is real"},"answer":null,"keyIdea":"When rows share a source (the same user, the same document, repeated runs of the same item) your effective sample size is closer to the number of sources than the number of rows, so aggregate to the source before you compute anything.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-interval-that-keeps-being-wrong","runs":[{"text":"The interval that keeps being wrong"}]},{"t":"p","runs":[{"text":"A team measures a support assistant on 500 logged conversations. The pass rate is 84%, the reported interval is plus or minus 3 points, and yet each fresh sample lands 6 or 7 points away. Nothing is broken in the arithmetic. The 500 conversations came from 40 customers, and rows from one customer resemble each other."}]},{"t":"p","runs":[{"text":"Every standard error formula assumes independent observations. Correlated rows carry less information than independent ones, and the amount less has a name."}]},{"t":"h2","id":"the-design-effect-in-numbers","runs":[{"text":"The design effect, in numbers"}]},{"t":"p","runs":[{"text":"With "},{"text":"m","code":true},{"text":" rows per cluster and an intra-cluster correlation "},{"text":"ICC","code":true},{"text":", the variance is inflated by:"}]},{"t":"code","lang":"","label":"Text","text":"design effect = 1 + (m - 1) * ICC"},{"t":"p","runs":[{"text":"For 500 rows from 40 customers, "},{"text":"m","code":true},{"text":" is 12.5. Suppose the ICC is 0.3, which is unremarkable for text from the same organisation:"}]},{"t":"code","lang":"","label":"Text","text":"design effect = 1 + 11.5 * 0.3 = 4.45\neffective n   = 500 / 4.45     ≈ 112"},{"t":"p","runs":[{"text":"You have 500 rows and about 112 rows’ worth of information. The honest interval is "},{"text":"sqrt(4.45)","code":true},{"text":", a bit over twice, wider than the one printed, which is precisely the discrepancy the team kept seeing."}]},{"t":"p","runs":[{"text":"The same arithmetic covers the other common case: 100 items run 5 times each at temperature 0.7. Those 500 rows are not 500 independent observations either, and the clusters are the items."}]},{"t":"h2","id":"three-fixes-in-order-of-preference","runs":[{"text":"Three fixes, in order of preference"}]},{"t":"p","runs":[{"text":"Aggregate to the cluster.","b":true},{"text":" Compute the rate per customer, then average across customers, and do all the statistics on those 40 numbers. It is simple, it is safe, and every formula becomes valid again because the 40 values are independent. The cost is a wider, correct interval."}]},{"t":"code","lang":"python","label":"Python","text":"per_customer = df.groupby(\"customer_id\")[\"passed\"].mean()\nn = len(per_customer)                               # 40, not 500\nse = per_customer.std(ddof=1) / np.sqrt(n)"},{"t":"p","runs":[{"text":"Cluster bootstrap.","b":true},{"text":" Resample customers with replacement, carrying all of each customer’s rows, and recompute. Use this when the statistic is not a simple mean: recall at 5, a weighted composite, a per-item pass rate you want to weight by volume."}]},{"t":"p","runs":[{"text":"Cluster-robust standard errors or a mixed model.","b":true},{"text":" "},{"text":"statsmodels","code":true},{"text":" will fit both. Worth it when you need to adjust for covariates at the same time; unnecessary otherwise."}]},{"t":"h2","id":"finding-the-clusters-which-are-rarely-labelled","runs":[{"text":"Finding the clusters, which are rarely labelled"}]},{"t":"p","runs":[{"text":"The customer column is the easy case. The clusters that cause trouble are the ones nobody wrote down:"}]},{"t":"ul","items":[[{"text":"One document, many questions.","b":true},{"text":" Forty questions generated from the same PDF share its formatting, its vocabulary and its ambiguities. The unit is the document."}],[{"text":"One template, many variants.","b":true},{"text":" Everything the previous module said about synthetic items applies here: forty rephrasings of one request are one observation wearing forty coats."}],[{"text":"One session, many turns.","b":true},{"text":" A conversation that goes wrong at turn two produces five failed turns, which is one failure."}],[{"text":"One time window.","b":true},{"text":" Everything logged during an outage shares the outage."}]]},{"t":"p","runs":[{"text":"Before computing an interval, ask what the rows have in common that the population would not. If the answer is “quite a lot”, find the grouping key and aggregate to it."}]},{"t":"h2","id":"repeated-runs-deserve-their-own-decomposition","runs":[{"text":"Repeated runs deserve their own decomposition"}]},{"t":"p","runs":[{"text":"When each item is run several times, two different variances are in play and they suggest different actions:"}]},{"t":"ul","items":[[{"text":"Item-to-item variance","b":true},{"text":": some items are harder. More runs will not reduce it; more "},{"text":"items","i":true},{"text":" will."}],[{"text":"Run-to-run variance","b":true},{"text":": the same item passing on Tuesday and failing on Wednesday. More items will not reduce it; more runs, a lower temperature, or a fixed seed will."}]]},{"t":"p","runs":[{"text":"Compute both. Take the per-item mean across runs, and look at the spread of those means (item variance) alongside the average spread within an item (run variance). If run-to-run spread is 5 points, then no amount of extra data makes a 3-point improvement detectable, and the correct next step is to make the system more deterministic rather than to label more."}]},{"t":"h2","id":"say-both-numbers-out-loud","runs":[{"text":"Say both numbers out loud"}]},{"t":"p","runs":[{"text":"The habit that prevents all of this is one line in every report: state the number of rows and the number of independent units."}]},{"t":"quote","runs":[{"text":"Pass rate 84% across 500 conversations from 40 customers (95% CI 76 to 92, computed at the customer level)."}]},{"t":"p","runs":[{"text":"A reader who sees “40 customers” immediately knows the resolution available. A reader who sees only “500 conversations” will believe the study is four times more precise than it is."}]},{"t":"h2","id":"the-limit-you-cannot-compute-your-way-out-of","runs":[{"text":"The limit you cannot compute your way out of"}],"role":"limit"},{"t":"p","runs":[{"text":"If the set contains 8 customers, no statistical technique makes it behave like 8,000. Clustering is a data-collection problem with a statistical symptom. When the effective sample is small, the truthful report says so and recommends widening collection (more customers, more documents, more languages) rather than quoting a narrow interval that the next sample will contradict."}]}],"check":{"question":[{"text":"An evaluation shows 84% on 500 conversations drawn from 40 customers. Which reported figure is most defensible?"}],"options":[{"text":[{"text":"84% with a 3-point interval, since 500 items is a large sample by evaluation standards."}],"correct":false,"tempting":[{"text":"Counts rows as independent observations; conversations from one customer are correlated, so 500 rows carry far less information than 500 independent draws."}]},{"text":[{"text":"84% with no interval, since clustered data cannot support a confidence interval."}],"correct":false,"tempting":[{"text":"Abandons the estimate rather than computing it at the right level; the customer-level mean supports a perfectly ordinary interval."}]},{"text":[{"text":"84% averaged across customers, with the interval computed from the spread of the 40 customer-level rates."}],"correct":true,"tempting":null},{"text":[{"text":"84% with the interval computed after removing the customers contributing the most conversations, so no single customer dominates."}],"correct":false,"tempting":[{"text":"Discarding data to reduce imbalance loses information and still treats the remaining rows as independent, which is the actual source of the error."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"bootstrap-pitfalls","title":"The bootstrap, and the three places it lies","minutes":9},"next":{"slug":"many-comparisons","title":"Twenty slices and one guaranteed surprise","minutes":8}}}