{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"how-many-clusters","title":"Choosing k, and why the elbow rarely helps","url":"https://addaly.com/learn/machine-learning-foundations/how-many-clusters","position":64,"of":92,"minutes":8,"answer":"The elbow rarely produces an unambiguous answer; use silhouette per cluster, the gap statistic, which can say that no clustering is right, and stability across subsamples.","keyIdea":"The elbow rarely produces an unambiguous answer; use silhouette per cluster, the gap statistic, which can say that no clustering is right, and stability across subsamples, and accept that the useful k is often set by what the organisation can act on."},"body":{"format":"markdown","text":"## The elbow method, and its problem\n\nPlot inertia against k. Inertia always falls as k rises (at k = n every point is its own cluster and inertia is zero) so you look for the \"elbow\" where the improvement slows.\n\nOn textbook data with well-separated blobs, the elbow is obvious. On real data it usually is not. The curve is a smooth decay with no clear corner, three people looking at the same plot pick three different values, and the method provides no way to adjudicate.\n\nIt is worth plotting, because it takes one line and occasionally the elbow is real. It is not worth treating as an answer.\n\n## Silhouette score\n\nA better measure. For each point, compute:\n\n- `a` = its mean distance to other points in its own cluster.\n- `b` = its mean distance to points in the nearest other cluster.\n- silhouette = `(b - a) / max(a, b)`.\n\nRuns from -1 to +1. Near +1 means the point is much closer to its own cluster than to any other. Near 0 means it sits on a boundary. Negative means it is closer to a different cluster than to its own: it is probably assigned wrongly.\n\n```python\nfrom sklearn.metrics import silhouette_score, silhouette_samples\nfor k in range(2, 11):\n    labels = KMeans(k, n_init=10, random_state=0).fit_predict(X)\n    print(k, round(silhouette_score(X, labels), 3))\n```\n\nThe mean silhouette gives a number per k that you can compare. As a rough guide, above 0.5 is a solid structure, 0.25 to 0.5 is weak, and below 0.25 means there is probably no cluster structure worth naming.\n\nBetter still, plot `silhouette_samples` per cluster rather than reading only the mean. A configuration with an excellent mean can contain one cluster where most points are near zero, and that cluster is the finding.\n\n## Other numerical criteria\n\n**Davies-Bouldin**: average similarity between each cluster and its most similar neighbour. Lower is better. Fast, and it also prefers spherical clusters.\n\n**Calinski-Harabasz**: ratio of between-cluster to within-cluster dispersion. Higher is better. Tends to favour larger k.\n\n**Gap statistic**: compares your inertia against the inertia obtained on uniformly random data with the same bounding box. This is the one with the most useful property: it can tell you that **k = 1 is best**, meaning there is no cluster structure at all. Every other method above assumes at least two clusters exist and never questions it.\n\nRun the gap statistic, or its poor-man's version (cluster a shuffled copy of your data and compare) on any clustering you plan to act on.\n\n## Stability, which is the most practical test\n\nA cluster structure that survives resampling is more likely to be real than one that does not.\n\nTake two random 80% subsets of your data, cluster both, and measure how much the assignments agree on the rows that appear in both, using the adjusted Rand index. Repeat for each candidate k.\n\n```python\nfrom sklearn.metrics import adjusted_rand_score\n```\n\nThe adjusted Rand index corrects for chance agreement, so 0 means random and 1 means identical. A k whose clusters reproduce at 0.85 across subsamples is telling you something. A k that reproduces at 0.3 is telling you the boundaries are arbitrary, whatever its silhouette says.\n\n> Figure: Two measures of k on the same customer table. Line chart of Score against k: Adjusted Rand index across two 80% subsamples (2, 0.6) (3, 0.72) (4, 0.88) (5, 0.61) (6, 0.5) (7, 0.45) (8, 0.42) (9, 0.4) (10, 0.38); Mean silhouette (2, 0.21) (3, 0.34) (4, 0.52) (5, 0.47) (6, 0.41) (7, 0.38) (8, 0.36) (9, 0.33) (10, 0.31). Both agree on k = 4: the mean silhouette clears 0.5 there and only there, and clusterings of two random 80 per cent subsamples agree at an adjusted Rand index of 0.88. At k = 5 the agreement collapses to 0.61, which says the fifth boundary is arbitrary whatever the silhouette says.\n\nThis is slower than the elbow and considerably more informative.\n\n## The answer nobody puts in the textbook\n\nOften k is decided by what happens next, not by the data.\n\nIf the marketing team can run four campaigns, k is 4. If the app has room for six recommendation shelves, k is 6. If a human has to write a description of each segment, k is under 10 because nobody will read twelve.\n\nThis is not a failure of rigour. Clustering is usually a tool for making a decision, and the decision has constraints the data does not know about. State the constraint, pick the k that satisfies it, and use the metrics to check whether the result is stable enough to rely on, which is a different and more answerable question than \"what is the true k\".\n\n## When to stop trying\n\nIf silhouette is 0.15 for every k, the gap statistic prefers k = 1, and subsampled clusterings disagree, the answer is that your data has no cluster structure. That is a legitimate finding and it is worth reporting as one. Continuing until a partition emerges will always succeed, and the result will describe your persistence rather than your customers."},"check":{"question":"A clustering has mean silhouette 0.61 at k=5, which looks strong. Clustering two 80% subsamples and comparing gives an adjusted Rand index of 0.28. What should the team conclude?","options":[{"text":"The silhouette is the more reliable measure since it uses all the data, so the clustering can be trusted.","correct":false,"tempting":"Uses more data but measures a different property: silhouette describes separation within one fitted partition, not whether that partition reappears on new data."},{"text":"The subsample size is too small at 80%, and repeating with 95% subsamples would raise the agreement.","correct":false,"tempting":"Larger subsamples overlap more and mechanically raise agreement, which is why 80% is the more informative test rather than a flaw to be tuned away."},{"text":"The high silhouette shows the clusters are compact and separated *as fitted*, but the low agreement across subsamples shows the boundaries move when the data changes, so the specific partition should not be treated as a real structure.","correct":true},{"text":"Adjusted Rand index below 0.5 always indicates a bug in the cluster-matching code rather than genuine instability.","correct":false,"tempting":"The adjusted index handles label permutation correctly, so a low value reflects genuine disagreement rather than mismatched cluster names."}]},"prev":{"slug":"clustering-with-k-means","title":"k-means, and what it assumes about your data","url":"https://addaly.com/learn/machine-learning-foundations/clustering-with-k-means"},"next":{"slug":"density-and-hierarchy","title":"DBSCAN and hierarchical clustering: when shape and structure matter","url":"https://addaly.com/learn/machine-learning-foundations/density-and-hierarchy"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Choosing k, and why the elbow rarely helps', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/how-many-clusters (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Choosing k, and why the elbow rarely helps","courseTitle":"Machine Learning, Foundations","module":{"index":7,"slug":"learning-without-labels","title":"Learning without labels, and learning a representation"},"answer":null,"keyIdea":"The elbow rarely produces an unambiguous answer; use silhouette per cluster, the gap statistic, which can say that no clustering is right, and stability across subsamples, and accept that the useful k is often set by what the organisation can act on.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Two measures of k on the same customer table","series":[{"label":"Adjusted Rand index across two 80% subsamples","points":[[2,0.6],[3,0.72],[4,0.88],[5,0.61],[6,0.5],[7,0.45],[8,0.42],[9,0.4],[10,0.38]]},{"label":"Mean silhouette","points":[[2,0.21],[3,0.34],[4,0.52],[5,0.47],[6,0.41],[7,0.38],[8,0.36],[9,0.33],[10,0.31]]}],"xLabel":"k","yLabel":"Score","caption":"Both agree on k = 4: the mean silhouette clears 0.5 there and only there, and clusterings of two random 80 per cent subsamples agree at an adjusted Rand index of 0.88. At k = 5 the agreement collapses to 0.61, which says the fifth boundary is arbitrary whatever the silhouette says."},"alt":"Figure: Two measures of k on the same customer table. Line chart of Score against k: Adjusted Rand index across two 80% subsamples (2, 0.6) (3, 0.72) (4, 0.88) (5, 0.61) (6, 0.5) (7, 0.45) (8, 0.42) (9, 0.4) (10, 0.38); Mean silhouette (2, 0.21) (3, 0.34) (4, 0.52) (5, 0.47) (6, 0.41) (7, 0.38) (8, 0.36) (9, 0.33) (10, 0.31). Both agree on k = 4: the mean silhouette clears 0.5 there and only there, and clusterings of two random 80 per cent subsamples agree at an adjusted Rand index of 0.88. At k = 5 the agreement collapses to 0.61, which says the fifth boundary is arbitrary whatever the silhouette says."}},"blocks":[{"t":"h2","id":"the-elbow-method-and-its-problem","runs":[{"text":"The elbow method, and its problem"}]},{"t":"p","runs":[{"text":"Plot inertia against k. Inertia always falls as k rises (at k = n every point is its own cluster and inertia is zero) so you look for the “elbow” where the improvement slows."}]},{"t":"p","runs":[{"text":"On textbook data with well-separated blobs, the elbow is obvious. On real data it usually is not. The curve is a smooth decay with no clear corner, three people looking at the same plot pick three different values, and the method provides no way to adjudicate."}]},{"t":"p","runs":[{"text":"It is worth plotting, because it takes one line and occasionally the elbow is real. It is not worth treating as an answer."}]},{"t":"h2","id":"silhouette-score","runs":[{"text":"Silhouette score"}]},{"t":"p","runs":[{"text":"A better measure. For each point, compute:"}]},{"t":"ul","items":[[{"text":"a","code":true},{"text":" = its mean distance to other points in its own cluster."}],[{"text":"b","code":true},{"text":" = its mean distance to points in the nearest other cluster."}],[{"text":"silhouette = "},{"text":"(b - a) / max(a, b)","code":true},{"text":"."}]]},{"t":"p","runs":[{"text":"Runs from -1 to +1. Near +1 means the point is much closer to its own cluster than to any other. Near 0 means it sits on a boundary. Negative means it is closer to a different cluster than to its own: it is probably assigned wrongly."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import silhouette_score, silhouette_samples\nfor k in range(2, 11):\n    labels = KMeans(k, n_init=10, random_state=0).fit_predict(X)\n    print(k, round(silhouette_score(X, labels), 3))"},{"t":"p","runs":[{"text":"The mean silhouette gives a number per k that you can compare. As a rough guide, above 0.5 is a solid structure, 0.25 to 0.5 is weak, and below 0.25 means there is probably no cluster structure worth naming."}]},{"t":"p","runs":[{"text":"Better still, plot "},{"text":"silhouette_samples","code":true},{"text":" per cluster rather than reading only the mean. A configuration with an excellent mean can contain one cluster where most points are near zero, and that cluster is the finding."}]},{"t":"h2","id":"other-numerical-criteria","runs":[{"text":"Other numerical criteria"}]},{"t":"p","runs":[{"text":"Davies-Bouldin","b":true},{"text":": average similarity between each cluster and its most similar neighbour. Lower is better. Fast, and it also prefers spherical clusters."}]},{"t":"p","runs":[{"text":"Calinski-Harabasz","b":true},{"text":": ratio of between-cluster to within-cluster dispersion. Higher is better. Tends to favour larger k."}]},{"t":"p","runs":[{"text":"Gap statistic","b":true},{"text":": compares your inertia against the inertia obtained on uniformly random data with the same bounding box. This is the one with the most useful property: it can tell you that "},{"text":"k = 1 is best","b":true},{"text":", meaning there is no cluster structure at all. Every other method above assumes at least two clusters exist and never questions it."}]},{"t":"p","runs":[{"text":"Run the gap statistic, or its poor-man’s version (cluster a shuffled copy of your data and compare) on any clustering you plan to act on."}]},{"t":"h2","id":"stability-which-is-the-most-practical-test","runs":[{"text":"Stability, which is the most practical test"}]},{"t":"p","runs":[{"text":"A cluster structure that survives resampling is more likely to be real than one that does not."}]},{"t":"p","runs":[{"text":"Take two random 80% subsets of your data, cluster both, and measure how much the assignments agree on the rows that appear in both, using the adjusted Rand index. Repeat for each candidate k."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import adjusted_rand_score"},{"t":"p","runs":[{"text":"The adjusted Rand index corrects for chance agreement, so 0 means random and 1 means identical. A k whose clusters reproduce at 0.85 across subsamples is telling you something. A k that reproduces at 0.3 is telling you the boundaries are arbitrary, whatever its silhouette says."}]},{"t":"p","runs":[{"text":"This is slower than the elbow and considerably more informative."}]},{"t":"h2","id":"the-answer-nobody-puts-in-the-textbook","runs":[{"text":"The answer nobody puts in the textbook"}]},{"t":"p","runs":[{"text":"Often k is decided by what happens next, not by the data."}]},{"t":"p","runs":[{"text":"If the marketing team can run four campaigns, k is 4. If the app has room for six recommendation shelves, k is 6. If a human has to write a description of each segment, k is under 10 because nobody will read twelve."}]},{"t":"p","runs":[{"text":"This is not a failure of rigour. Clustering is usually a tool for making a decision, and the decision has constraints the data does not know about. State the constraint, pick the k that satisfies it, and use the metrics to check whether the result is stable enough to rely on, which is a different and more answerable question than “what is the true k”."}]},{"t":"h2","id":"when-to-stop-trying","runs":[{"text":"When to stop trying"}]},{"t":"p","runs":[{"text":"If silhouette is 0.15 for every k, the gap statistic prefers k = 1, and subsampled clusterings disagree, the answer is that your data has no cluster structure. That is a legitimate finding and it is worth reporting as one. Continuing until a partition emerges will always succeed, and the result will describe your persistence rather than your customers."}]}],"check":{"question":[{"text":"A clustering has mean silhouette 0.61 at k=5, which looks strong. Clustering two 80% subsamples and comparing gives an adjusted Rand index of 0.28. What should the team conclude?"}],"options":[{"text":[{"text":"The silhouette is the more reliable measure since it uses all the data, so the clustering can be trusted."}],"correct":false,"tempting":[{"text":"Uses more data but measures a different property: silhouette describes separation within one fitted partition, not whether that partition reappears on new data."}]},{"text":[{"text":"The subsample size is too small at 80%, and repeating with 95% subsamples would raise the agreement."}],"correct":false,"tempting":[{"text":"Larger subsamples overlap more and mechanically raise agreement, which is why 80% is the more informative test rather than a flaw to be tuned away."}]},{"text":[{"text":"The high silhouette shows the clusters are compact and separated "},{"text":"as fitted","i":true},{"text":", but the low agreement across subsamples shows the boundaries move when the data changes, so the specific partition should not be treated as a real structure."}],"correct":true,"tempting":null},{"text":[{"text":"Adjusted Rand index below 0.5 always indicates a bug in the cluster-matching code rather than genuine instability."}],"correct":false,"tempting":[{"text":"The adjusted index handles label permutation correctly, so a low value reflects genuine disagreement rather than mismatched cluster names."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"clustering-with-k-means","title":"k-means, and what it assumes about your data","minutes":9},"next":{"slug":"density-and-hierarchy","title":"DBSCAN and hierarchical clustering: when shape and structure matter","minutes":8}}}