Machine Learning, Foundations · Chapter 07
How many clusters, really
1:55 · Narrated with a synthetic voice. Captions are available in the player.
Read the transcript
Clustering will always give you an answer, including from pure noise. The elbow plot rarely produces one anybody can defend. Silhouette is better, and you should read it per cluster rather than as a single mean. The gap statistic is the one measure that can tell you no clustering is appropriate at all, because it compares your result against random data with the same bounding box. And the most practical test is stability. Cluster two eighty per cent subsets and see how far the assignments agree. A structure that reproduces at zero point eight five is telling you something real. One that reproduces at zero point three is telling you about your own persistence. An embedding maps things to vectors so that geometric closeness means similarity in whatever sense the training task defined. Pretrained ones are free, small enough to run on an ordinary laptop processor, and good enough for most applications. Nothing has to leave the machine, which matters when the documents are confidential. Two warnings from practice. Chunking decides more than the choice of model does: too large and one vector averages several topics, too small and the text loses the context that made it interpretable. And pure embedding search fails on order numbers, on part codes and on negation, which is why production systems run keyword search alongside it and combine the two rankings.