Five hundred rows, forty users

The interval that keeps being wrong

A team measures a support assistant on 500 logged conversations. The pass rate is 84%, the reported interval is plus or minus 3 points, and yet each fresh sample lands 6 or 7 points away. Nothing is broken in the arithmetic. The 500 conversations came from 40 customers, and rows from one customer resemble each other.

Every standard error formula assumes independent observations. Correlated rows carry less information than independent ones, and the amount less has a name.

The design effect, in numbers

With m rows per cluster and an intra-cluster correlation ICC, the variance is inflated by:

design effect = 1 + (m - 1) * ICC

For 500 rows from 40 customers, m is 12.5. Suppose the ICC is 0.3, which is unremarkable for text from the same organisation:

design effect = 1 + 11.5 * 0.3 = 4.45
effective n   = 500 / 4.45     ≈ 112

You have 500 rows and about 112 rows’ worth of information. The honest interval is sqrt(4.45), a bit over twice, wider than the one printed, which is precisely the discrepancy the team kept seeing.

The same arithmetic covers the other common case: 100 items run 5 times each at temperature 0.7. Those 500 rows are not 500 independent observations either, and the clusters are the items.

Three fixes, in order of preference

Aggregate to the cluster. Compute the rate per customer, then average across customers, and do all the statistics on those 40 numbers. It is simple, it is safe, and every formula becomes valid again because the 40 values are independent. The cost is a wider, correct interval.

per_customer = df.groupby("customer_id")["passed"].mean()
n = len(per_customer)                               # 40, not 500
se = per_customer.std(ddof=1) / np.sqrt(n)

Cluster bootstrap. Resample customers with replacement, carrying all of each customer’s rows, and recompute. Use this when the statistic is not a simple mean: recall at 5, a weighted composite, a per-item pass rate you want to weight by volume.

Cluster-robust standard errors or a mixed model. statsmodels will fit both. Worth it when you need to adjust for covariates at the same time; unnecessary otherwise.

Finding the clusters, which are rarely labelled

The customer column is the easy case. The clusters that cause trouble are the ones nobody wrote down:

• One document, many questions. Forty questions generated from the same PDF share its formatting, its vocabulary and its ambiguities. The unit is the document.
• One template, many variants. Everything the previous module said about synthetic items applies here: forty rephrasings of one request are one observation wearing forty coats.
• One session, many turns. A conversation that goes wrong at turn two produces five failed turns, which is one failure.
• One time window. Everything logged during an outage shares the outage.

Before computing an interval, ask what the rows have in common that the population would not. If the answer is “quite a lot”, find the grouping key and aggregate to it.

Repeated runs deserve their own decomposition

When each item is run several times, two different variances are in play and they suggest different actions:

• Item-to-item variance: some items are harder. More runs will not reduce it; more items will.
• Run-to-run variance: the same item passing on Tuesday and failing on Wednesday. More items will not reduce it; more runs, a lower temperature, or a fixed seed will.

Compute both. Take the per-item mean across runs, and look at the spread of those means (item variance) alongside the average spread within an item (run variance). If run-to-run spread is 5 points, then no amount of extra data makes a 3-point improvement detectable, and the correct next step is to make the system more deterministic rather than to label more.

Say both numbers out loud

The habit that prevents all of this is one line in every report: state the number of rows and the number of independent units.

Pass rate 84% across 500 conversations from 40 customers (95% CI 76 to 92, computed at the customer level).

A reader who sees “40 customers” immediately knows the resolution available. A reader who sees only “500 conversations” will believe the study is four times more precise than it is.

The limit you cannot compute your way out of

If the set contains 8 customers, no statistical technique makes it behave like 8,000. Clustering is a data-collection problem with a statistical symptom. When the effective sample is small, the truthful report says so and recommends widening collection (more customers, more documents, more languages) rather than quoting a narrow interval that the next sample will contradict.