Choosing k, and why the elbow rarely helps

The elbow method, and its problem

Plot inertia against k. Inertia always falls as k rises (at k = n every point is its own cluster and inertia is zero) so you look for the “elbow” where the improvement slows.

On textbook data with well-separated blobs, the elbow is obvious. On real data it usually is not. The curve is a smooth decay with no clear corner, three people looking at the same plot pick three different values, and the method provides no way to adjudicate.

It is worth plotting, because it takes one line and occasionally the elbow is real. It is not worth treating as an answer.

Silhouette score

A better measure. For each point, compute:

• a = its mean distance to other points in its own cluster.
• b = its mean distance to points in the nearest other cluster.
• silhouette = (b - a) / max(a, b).

Runs from -1 to +1. Near +1 means the point is much closer to its own cluster than to any other. Near 0 means it sits on a boundary. Negative means it is closer to a different cluster than to its own: it is probably assigned wrongly.

from sklearn.metrics import silhouette_score, silhouette_samples
for k in range(2, 11):
    labels = KMeans(k, n_init=10, random_state=0).fit_predict(X)
    print(k, round(silhouette_score(X, labels), 3))

The mean silhouette gives a number per k that you can compare. As a rough guide, above 0.5 is a solid structure, 0.25 to 0.5 is weak, and below 0.25 means there is probably no cluster structure worth naming.

Better still, plot silhouette_samples per cluster rather than reading only the mean. A configuration with an excellent mean can contain one cluster where most points are near zero, and that cluster is the finding.

Other numerical criteria

Davies-Bouldin: average similarity between each cluster and its most similar neighbour. Lower is better. Fast, and it also prefers spherical clusters.

Calinski-Harabasz: ratio of between-cluster to within-cluster dispersion. Higher is better. Tends to favour larger k.

Gap statistic: compares your inertia against the inertia obtained on uniformly random data with the same bounding box. This is the one with the most useful property: it can tell you that k = 1 is best, meaning there is no cluster structure at all. Every other method above assumes at least two clusters exist and never questions it.

Run the gap statistic, or its poor-man’s version (cluster a shuffled copy of your data and compare) on any clustering you plan to act on.

Stability, which is the most practical test

A cluster structure that survives resampling is more likely to be real than one that does not.

Take two random 80% subsets of your data, cluster both, and measure how much the assignments agree on the rows that appear in both, using the adjusted Rand index. Repeat for each candidate k.

from sklearn.metrics import adjusted_rand_score

The adjusted Rand index corrects for chance agreement, so 0 means random and 1 means identical. A k whose clusters reproduce at 0.85 across subsamples is telling you something. A k that reproduces at 0.3 is telling you the boundaries are arbitrary, whatever its silhouette says.

This is slower than the elbow and considerably more informative.

The answer nobody puts in the textbook

Often k is decided by what happens next, not by the data.

If the marketing team can run four campaigns, k is 4. If the app has room for six recommendation shelves, k is 6. If a human has to write a description of each segment, k is under 10 because nobody will read twelve.

This is not a failure of rigour. Clustering is usually a tool for making a decision, and the decision has constraints the data does not know about. State the constraint, pick the k that satisfies it, and use the metrics to check whether the result is stable enough to rely on, which is a different and more answerable question than “what is the true k”.

When to stop trying

If silhouette is 0.15 for every k, the gap statistic prefers k = 1, and subsampled clusterings disagree, the answer is that your data has no cluster structure. That is a legitimate finding and it is worth reporting as one. Continuing until a partition emerges will always succeed, and the result will describe your persistence rather than your customers.