t-SNE and UMAP, and how they mislead

A different goal from PCA

PCA preserves global structure: distances and directions across the whole dataset, as well as a linear projection can. It is often disappointing for visualisation, because two dimensions of a 300-dimensional dataset usually look like a blob.

t-SNE and UMAP have a different objective: preserve local neighbourhoods. Points that were close in the original space should be close in the plot. Points that were far can go anywhere.

That trade produces much more revealing pictures, and it is also the source of every way these plots deceive people.

from sklearn.manifold import TSNE
emb = TSNE(n_components=2, perplexity=30, init="pca",
           random_state=0).fit_transform(X_reduced)

Four things the plot does not tell you

1. Cluster sizes are meaningless. t-SNE expands dense regions and contracts sparse ones, deliberately, so that dense clusters do not collapse into dots. A large blob is not a large cluster.

2. Distances between clusters are meaningless. Two clusters at opposite ends of the plot are not more different than two adjacent ones. Only the fact that points within a cluster are neighbours is reliable.

3. Apparent clusters can be artefacts. Run t-SNE on uniform random noise with a low perplexity and it produces convincing-looking clusters. This is not a rare pathology; it happens readily. Always run the same procedure on shuffled data before believing a structure you see.

4. The result changes with the seed and the parameters. perplexity roughly sets how many neighbours each point tries to stay near, and values of 5, 30 and 100 on the same data can give visibly different pictures. Run several and report what is stable across them.

UMAP

UMAP does a similar job with a different mathematical construction and three practical advantages: it is much faster on large data, it preserves more global structure than t-SNE (though still not enough to trust distances), and it can transform new points into an existing embedding, which t-SNE cannot do at all.

import umap
reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=0)
emb = reducer.fit_transform(X_reduced)

n_neighbors trades local against global structure: low values emphasise fine local detail, high values the broad shape. min_dist controls how tightly points may pack, which affects appearance more than substance.

UMAP is a free package and is now the more common choice. Its global structure is better than t-SNE’s and still not a metric you should quote.

Run PCA first

Both methods are slow and both suffer in very high dimensions. The standard practice is to reduce to 30 to 50 components with PCA and then run t-SNE or UMAP on that. It is faster, it removes some noise, and it usually improves the picture.

What these are legitimately for

• Looking at your data before modelling. If the classes visibly separate, a simple model will probably work. If they are thoroughly mixed, expect a hard problem, although note the converse does not hold, since a projection that fails to separate them does not prove no model can.
• Finding data problems. Duplicate rows appear as unnaturally tight points. A mislabelled group shows up as a coloured island inside another cluster. Batch effects (data collected on different days or instruments) appear as separation by batch rather than by anything you care about, and this is one of the most valuable things these plots reveal.
• Inspecting embeddings. Colour points by a known attribute and see whether the representation has organised itself by it.

What they are not for

They are not a preprocessing step for a supervised model. The output coordinates have no stable meaning, t-SNE cannot transform new data at all, and UMAP’s transform is not guaranteed stable enough to serve as production features. Use PCA if you need a reduction that a model will consume.

And they are not evidence. A convincing t-SNE plot is a hypothesis. Confirm it with a number (a silhouette score, a classifier that can tell the groups apart, a stability check) before it appears in a document that someone will act on.

The cheapest confirmation is usually a classifier. If you believe the plot shows two real groups, label the points by which group they appear in and train a simple model to predict that label from the original features, evaluated on held-out rows. A group that a logistic regression can recover at 0.9 AUC from three interpretable columns is a real and describable structure. One that no model can recover above chance was a shape in the projection and nothing more.