{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"comparing-two-models","title":"Is that improvement real?","url":"https://addaly.com/learn/machine-learning-foundations/comparing-two-models","position":48,"of":92,"minutes":9,"answer":"Bootstrap the difference between two models on the same resampled rows and report the interval; a t-test across cross-validation folds violates independence and declares significance far too often.","keyIdea":"Bootstrap the difference between two models on the same resampled rows and report the interval; a t-test across cross-validation folds violates independence and declares significance far too often."},"body":{"format":"markdown","text":"## The situation\n\nModel A scores 0.847. Model B scores 0.852. Do you ship B?\n\nThe wrong answer is \"yes, it is higher\". Both numbers were measured on a finite sample, both carry uncertainty, and 0.005 may be well inside it.\n\n## The bootstrap, which is all you need\n\nResample your evaluation set with replacement, recompute the metric, repeat a thousand times. The spread of those thousand values is the sampling distribution of your score.\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\nn = len(y_test)\ndiffs = []\nfor _ in range(2000):\n    idx = rng.integers(0, n, n)\n    diffs.append(roc_auc_score(y_test[idx], pB[idx])\n                 - roc_auc_score(y_test[idx], pA[idx]))\nlo, hi = np.percentile(diffs, [2.5, 97.5])\nprint(f\"difference {np.mean(diffs):.4f}  95% CI [{lo:.4f}, {hi:.4f}]\")\n```\n\nOne detail carries most of the value: **resample the rows once and score both models on the same resampled rows.** This is a paired comparison, and it removes the variance caused by some rows being harder. An unpaired comparison of two intervals is far less sensitive and will tell you \"inconclusive\" when the paired version would have given you a clear answer.\n\nIf the interval on the difference contains zero, you do not have evidence that B is better. That is not the same as evidence they are equal; it means your test set cannot resolve a difference this small.\n\n## McNemar's test, for two classifiers on the same data\n\nWhen comparing predicted labels rather than scores, the relevant information is only in the rows where the two models disagree.\n\n- `b` = rows A got right and B got wrong.\n- `c` = rows A got wrong and B got right.\n\nRows where both were right or both were wrong tell you nothing about which is better. Under the hypothesis that the models are equally good, `b` and `c` should be about equal, and McNemar's test asks how surprising the observed imbalance is.\n\n```python\nfrom statsmodels.stats.contingency_tables import mcnemar\nmcnemar([[both_right, b], [c, both_wrong]], exact=True)\n```\n\nIf B beat A on 60 rows and lost on 45, that is a 60-45 split out of 105 disagreements: nowhere near significant. If it is 60-20, it is. The absolute accuracy difference can look identical in both cases, which is why the disagreement counts are the thing to look at.\n\n## Cross-validation is not a free significance test\n\nA tempting move: run 10-fold cross-validation on both models and run a t-test on the ten pairs of scores.\n\nDo not. The folds share training data, so the ten differences are correlated, and the standard t-test assumes independence. Dietterich showed in 1998 that this procedure has a badly inflated false positive rate: it declares differences significant far more often than it should. Repeated cross-validation makes it worse, not better, because the extra repeats are even more correlated.\n\nWhat to do instead: use a held-out test set and bootstrap it, which is what the first section does. Or report the fold-to-fold spread as a description of stability without attaching a p-value to it.\n\n## How large a difference can you even see?\n\nWork it out before running the comparison. On a 2,000-row test set with accuracy near 85%, the standard error is about 0.8 points, and the standard error on a *paired difference* is smaller: often around 0.3 to 0.5 points depending on how correlated the models are.\n\nSo a 0.2-point improvement is not measurable on that set, no matter how many times you rerun it. Knowing this in advance saves you from a week of tuning aimed at a difference your evaluation cannot detect.\n\n## Practical significance\n\nThe last step is the one that gets skipped. Suppose the improvement is real: 0.847 to 0.852, statistically solid on a large test set.\n\nIs it worth shipping? Model B is a stacked ensemble that takes four hours to train, 300ms to serve instead of 12ms, and cannot be explained to a regulator. The gain is 0.5% of AUC, which might translate to a handful of extra fraud cases caught per month.\n\nStatistical significance is a statement about whether an effect exists. Whether it matters is a separate judgement involving latency, cost, maintenance and explainability, and the person who can hold both is considerably more useful than the one who can only compute the first."},"check":{"question":"Two classifiers are compared on 5,000 test rows. They agree on 4,700. Of the 300 disagreements, model B is right on 170 and model A on 130. What does McNemar's framing say?","options":[{"text":"B is clearly better, since it wins 57% of the contested cases across a large test set.","correct":false,"tempting":"Treats a 57% share as decisive, but a 170-130 split of 300 is close to what chance produces and does not reach conventional significance."},{"text":"The comparison is invalid because the 4,700 agreements should also be counted, and they show the models are near-identical.","correct":false,"tempting":"The agreements are correctly excluded: rows where both models behave the same carry no information about which is better."},{"text":"A 170-130 split of 300 disagreements is within the range chance produces, so the test set does not provide evidence that B is better despite the 0.8-point accuracy gap.","correct":true},{"text":"McNemar's test cannot be applied because the models were evaluated on the same rows rather than independent samples.","correct":false,"tempting":"Inverts the requirement: McNemar's test exists precisely for paired predictions on the same rows."}]},"prev":{"slug":"regression-metrics","title":"Metrics for a number, and what each one hides","url":"https://addaly.com/learn/machine-learning-foundations/regression-metrics"},"next":{"slug":"cost-sensitive-decisions","title":"From a probability to a decision worth making","url":"https://addaly.com/learn/machine-learning-foundations/cost-sensitive-decisions"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Is that improvement real?', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/comparing-two-models (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Is that improvement real?","courseTitle":"Machine Learning, Foundations","module":{"index":5,"slug":"measuring-honestly","title":"Measuring a model honestly"},"answer":null,"keyIdea":"Bootstrap the difference between two models on the same resampled rows and report the interval; a t-test across cross-validation folds violates independence and declares significance far too often.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-situation","runs":[{"text":"The situation"}]},{"t":"p","runs":[{"text":"Model A scores 0.847. Model B scores 0.852. Do you ship B?"}]},{"t":"p","runs":[{"text":"The wrong answer is “yes, it is higher”. Both numbers were measured on a finite sample, both carry uncertainty, and 0.005 may be well inside it."}]},{"t":"h2","id":"the-bootstrap-which-is-all-you-need","runs":[{"text":"The bootstrap, which is all you need"}]},{"t":"p","runs":[{"text":"Resample your evaluation set with replacement, recompute the metric, repeat a thousand times. The spread of those thousand values is the sampling distribution of your score."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nrng = np.random.default_rng(0)\nn = len(y_test)\ndiffs = []\nfor _ in range(2000):\n    idx = rng.integers(0, n, n)\n    diffs.append(roc_auc_score(y_test[idx], pB[idx])\n                 - roc_auc_score(y_test[idx], pA[idx]))\nlo, hi = np.percentile(diffs, [2.5, 97.5])\nprint(f\"difference {np.mean(diffs):.4f}  95% CI [{lo:.4f}, {hi:.4f}]\")"},{"t":"p","runs":[{"text":"One detail carries most of the value: "},{"text":"resample the rows once and score both models on the same resampled rows.","b":true},{"text":" This is a paired comparison, and it removes the variance caused by some rows being harder. An unpaired comparison of two intervals is far less sensitive and will tell you “inconclusive” when the paired version would have given you a clear answer."}]},{"t":"p","runs":[{"text":"If the interval on the difference contains zero, you do not have evidence that B is better. That is not the same as evidence they are equal; it means your test set cannot resolve a difference this small."}]},{"t":"h2","id":"mcnemar-s-test-for-two-classifiers-on-the-same-data","runs":[{"text":"McNemar’s test, for two classifiers on the same data"}]},{"t":"p","runs":[{"text":"When comparing predicted labels rather than scores, the relevant information is only in the rows where the two models disagree."}]},{"t":"ul","items":[[{"text":"b","code":true},{"text":" = rows A got right and B got wrong."}],[{"text":"c","code":true},{"text":" = rows A got wrong and B got right."}]]},{"t":"p","runs":[{"text":"Rows where both were right or both were wrong tell you nothing about which is better. Under the hypothesis that the models are equally good, "},{"text":"b","code":true},{"text":" and "},{"text":"c","code":true},{"text":" should be about equal, and McNemar’s test asks how surprising the observed imbalance is."}]},{"t":"code","lang":"python","label":"Python","text":"from statsmodels.stats.contingency_tables import mcnemar\nmcnemar([[both_right, b], [c, both_wrong]], exact=True)"},{"t":"p","runs":[{"text":"If B beat A on 60 rows and lost on 45, that is a 60-45 split out of 105 disagreements: nowhere near significant. If it is 60-20, it is. The absolute accuracy difference can look identical in both cases, which is why the disagreement counts are the thing to look at."}]},{"t":"h2","id":"cross-validation-is-not-a-free-significance-test","runs":[{"text":"Cross-validation is not a free significance test"}]},{"t":"p","runs":[{"text":"A tempting move: run 10-fold cross-validation on both models and run a t-test on the ten pairs of scores."}]},{"t":"p","runs":[{"text":"Do not. The folds share training data, so the ten differences are correlated, and the standard t-test assumes independence. Dietterich showed in 1998 that this procedure has a badly inflated false positive rate: it declares differences significant far more often than it should. Repeated cross-validation makes it worse, not better, because the extra repeats are even more correlated."}]},{"t":"p","runs":[{"text":"What to do instead: use a held-out test set and bootstrap it, which is what the first section does. Or report the fold-to-fold spread as a description of stability without attaching a p-value to it."}]},{"t":"h2","id":"how-large-a-difference-can-you-even-see","runs":[{"text":"How large a difference can you even see?"}]},{"t":"p","runs":[{"text":"Work it out before running the comparison. On a 2,000-row test set with accuracy near 85%, the standard error is about 0.8 points, and the standard error on a "},{"text":"paired difference","i":true},{"text":" is smaller: often around 0.3 to 0.5 points depending on how correlated the models are."}]},{"t":"p","runs":[{"text":"So a 0.2-point improvement is not measurable on that set, no matter how many times you rerun it. Knowing this in advance saves you from a week of tuning aimed at a difference your evaluation cannot detect."}]},{"t":"h2","id":"practical-significance","runs":[{"text":"Practical significance"}]},{"t":"p","runs":[{"text":"The last step is the one that gets skipped. Suppose the improvement is real: 0.847 to 0.852, statistically solid on a large test set."}]},{"t":"p","runs":[{"text":"Is it worth shipping? Model B is a stacked ensemble that takes four hours to train, 300ms to serve instead of 12ms, and cannot be explained to a regulator. The gain is 0.5% of AUC, which might translate to a handful of extra fraud cases caught per month."}]},{"t":"p","runs":[{"text":"Statistical significance is a statement about whether an effect exists. Whether it matters is a separate judgement involving latency, cost, maintenance and explainability, and the person who can hold both is considerably more useful than the one who can only compute the first."}]}],"check":{"question":[{"text":"Two classifiers are compared on 5,000 test rows. They agree on 4,700. Of the 300 disagreements, model B is right on 170 and model A on 130. What does McNemar’s framing say?"}],"options":[{"text":[{"text":"B is clearly better, since it wins 57% of the contested cases across a large test set."}],"correct":false,"tempting":[{"text":"Treats a 57% share as decisive, but a 170-130 split of 300 is close to what chance produces and does not reach conventional significance."}]},{"text":[{"text":"The comparison is invalid because the 4,700 agreements should also be counted, and they show the models are near-identical."}],"correct":false,"tempting":[{"text":"The agreements are correctly excluded: rows where both models behave the same carry no information about which is better."}]},{"text":[{"text":"A 170-130 split of 300 disagreements is within the range chance produces, so the test set does not provide evidence that B is better despite the 0.8-point accuracy gap."}],"correct":true,"tempting":null},{"text":[{"text":"McNemar’s test cannot be applied because the models were evaluated on the same rows rather than independent samples."}],"correct":false,"tempting":[{"text":"Inverts the requirement: McNemar’s test exists precisely for paired predictions on the same rows."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"regression-metrics","title":"Metrics for a number, and what each one hides","minutes":8},"next":{"slug":"cost-sensitive-decisions","title":"From a probability to a decision worth making","minutes":8}}}