The classic algorithms, and when each is right

A short catalogue

Eight families cover almost everything built before deep learning, and most of what is still built on tables today. Each is a different answer to "what shape may the function take".

Linear and logistic regression. Weighted sum of the inputs. Linear regression predicts a number; logistic regression squashes the sum into a probability. Small, fast, and each coefficient is a readable statement: this feature moves the log-odds by this much. Cannot represent interactions or curves unless you build them in as features. Use it as your floor — the score every other model must beat to justify itself — and as your answer whenever you must explain a decision to a regulator or a customer.

k-nearest neighbours. No training. To predict, find the k most similar rows you have seen and take their average or majority vote. Everything hinges on the distance function, so features must be scaled and irrelevant columns actively hurt. Falls apart in high dimensions, where everything is roughly equidistant from everything. Good when you have few examples, a distance that genuinely means something, and you want a baseline in ten minutes.

Decision trees. A learned sequence of yes/no questions. "Is income above ₹40,000? Then is tenure above 8 months?" Handles non-linearity and interactions for free, needs no scaling, and takes mixed data types. A single tree is readable enough to print and put in front of a domain expert, which is sometimes the entire reason to use one. On its own it is unstable — change 5% of the data and the tree can be different — and it overfits unless pruned.

Random forest. Hundreds of trees, each on a bootstrap sample of rows and a random subset of features, averaged. Averaging cancels the instability. Very hard to make it perform badly, robust to defaults, and the honest workhorse when you do not want to think about hyperparameters.

Gradient boosting (XGBoost, LightGBM, CatBoost). Trees again, but built in sequence, each one fitted to the errors the previous ones left behind. Consistently the strongest thing on tabular data and has been for a decade. The cost is hyperparameters that matter — learning rate, depth, number of trees, regularization — and a genuine ability to overfit if you set them carelessly.

Support vector machines. Find the boundary with the widest margin between classes; the kernel trick lets that boundary be curved. Excellent when you have many features and few rows, which is why they still appear in genomics and other wide-data fields. Training scales badly past tens of thousands of rows, and the output is not a probability without extra work.

Naive Bayes. Applies Bayes' rule while pretending every feature is independent, which is false. It is fast, needs very little data, and works better than the assumption deserves for text. Its probabilities are badly calibrated — treat them as a ranking, never as a number.

k-means. Unsupervised. Partition rows into k groups by distance to k moving centres. Cheap and useful for exploration. You must choose k, it assumes roughly round and similarly sized clusters, and it returns clusters whether or not any exist.

The default

For a table of rows and columns:

1. 1Fit logistic or linear regression. Record the score. That is your floor.
2. 2Fit gradient boosting with sensible defaults. That is usually your ceiling.
3. 3If the gap between them is small, your features are the constraint. Go back to Lesson 3.
4. 4Reach for anything else only for a stated reason: a printable tree because a committee must read it, an SVM because you have 300 rows and 8,000 columns, k-NN because a good distance is the whole domain.

Deep learning does not belong in that list for tabular data. Repeated benchmarks find boosted trees ahead of neural networks on medium-sized tables, and the neural network costs more to tune and run. Use networks where they win: images, audio, text, and anything sequential.

What the gap between two models tells you

This is the useful move, and it costs an afternoon.

Logistic regression can only draw a straight boundary. Gradient boosting can carve arbitrary regions and capture interactions between features. If you fit both and the scores are within a point of each other, that is evidence: there is little non-linear structure left for a flexible model to find, so the ceiling is set by what your columns contain.

The reflex at that point is to reach for something more powerful still. It is the wrong reflex. Two model families with very different expressive power agreeing on a number is a message about the data, and the answer is a better feature, a better label, or a column you do not yet have.