{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"the-classic-algorithms","title":"The classic algorithms, and when each is right","url":"https://addaly.com/learn/machine-learning-foundations/the-classic-algorithms","position":53,"of":92,"minutes":10,"answer":"On tabular data, use linear models as a floor and gradient boosting as the answer; everything else needs a stated reason.","keyIdea":"On tabular data, use linear models as a floor and gradient boosting as the answer; everything else needs a stated reason."},"body":{"format":"markdown","text":"## A short catalogue\n\nEight families cover almost everything built before deep learning, and most of what is still built on tables today. Each is a different answer to \"what shape may the function take\".\n\n**Linear and logistic regression.** Weighted sum of the inputs. Linear regression predicts a number; logistic regression squashes the sum into a probability. Small, fast, and each coefficient is a readable statement: this feature moves the log-odds by this much. Cannot represent interactions or curves unless you build them in as features. Use it as your floor (the score every other model must beat to justify itself) and as your answer whenever you must explain a decision to a regulator or a customer.\n\n**k-nearest neighbours.** No training. To predict, find the k most similar rows you have seen and take their average or majority vote. Everything hinges on the distance function, so features must be scaled and irrelevant columns actively hurt. Falls apart in high dimensions, where everything is roughly equidistant from everything. Good when you have few examples, a distance that means something, and you want a baseline in ten minutes.\n\n**Decision trees.** A learned sequence of yes/no questions. \"Is income above ₹40,000? Then is tenure above 8 months?\" Handles non-linearity and interactions for free, needs no scaling, and takes mixed data types. A single tree is readable enough to print and put in front of a domain expert, which is sometimes the entire reason to use one. On its own it is unstable (change 5% of the data and the tree can be different) and it overfits unless pruned.\n\n**Random forest.** Hundreds of trees, each on a bootstrap sample of rows and a random subset of features, averaged. Averaging cancels the instability. Very hard to make it perform badly, robust to defaults, and the honest workhorse when you do not want to think about hyperparameters.\n\n**Gradient boosting** (XGBoost, LightGBM, CatBoost). Trees again, but built in sequence, each one fitted to the errors the previous ones left behind. Consistently the strongest thing on tabular data and has been for a decade. The cost is hyperparameters that matter (learning rate, depth, number of trees, regularization) and a genuine ability to overfit if you set them carelessly.\n\n**Support vector machines.** Find the boundary with the widest margin between classes; the kernel trick lets that boundary be curved. Excellent when you have many features and few rows, which is why they still appear in genomics and other wide-data fields. Training scales badly past tens of thousands of rows, and the output is not a probability without extra work.\n\n**Naive Bayes.** Applies Bayes' rule while pretending every feature is independent, which is false. It is fast, needs very little data, and works better than the assumption deserves for text. Its probabilities are badly calibrated: treat them as a ranking, never as a number.\n\n**k-means.** Unsupervised. Partition rows into k groups by distance to k moving centres. Cheap and useful for exploration. You must choose k, it assumes roughly round and similarly sized clusters, and it returns clusters whether or not any exist.\n\n## The default\n\nFor a table of rows and columns:\n\n1. Fit logistic or linear regression. Record the score. That is your floor.\n2. Fit gradient boosting with sensible defaults. That is usually your ceiling.\n3. If the gap between them is small, your features are the constraint. Go back to Lesson 3.\n4. Reach for anything else only for a stated reason: a printable tree because a committee must read it, an SVM because you have 300 rows and 8,000 columns, k-NN because a good distance is the whole domain.\n\nDeep learning does not belong in that list for tabular data. Repeated benchmarks find boosted trees ahead of neural networks on medium-sized tables, and the neural network costs more to tune and run. Use networks where they win: images, audio, text, and anything sequential.\n\n## What the gap between two models tells you\n\nThis is the useful move, and it costs an afternoon.\n\nLogistic regression can only draw a straight boundary. Gradient boosting can carve arbitrary regions and capture interactions between features. If you fit both and the scores are within a point of each other, that is evidence: there is little non-linear structure left for a flexible model to find, so the ceiling is set by what your columns contain.\n\nThe reflex at that point is to reach for something more powerful still. It is the wrong reflex. Two model families with very different expressive power agreeing on a number is a message about the data, and the answer is a better feature, a better label, or a column you do not yet have."},"check":{"question":"A team predicts crop yield from soil and weather columns. Their logistic-style linear model scores poorly. They switch to a random forest and the score moves by half a point. What does that most likely tell them?","options":[{"text":"The columns they have do not carry much information about yield.","correct":true},{"text":"Random forests are not strong enough; they should try gradient boosting or a neural network next.","correct":false,"tempting":"There is a real ladder of model power, so a flat result reads naturally as not having climbed high enough yet."},{"text":"The two models are broadly similar, so a similar score is expected and uninformative.","correct":false,"tempting":"Both count as standard ML models, but a forest can express curves and interactions that a linear model structurally cannot, which is exactly why their agreement carries information."},{"text":"They need more training rows.","correct":false,"tempting":"More data reliably fixes variance problems, but here two model families with very different capacity land in the same place, which points at the inputs rather than the sample size."}]},"prev":{"slug":"slice-based-error-analysis","title":"The average is hiding the failure","url":"https://addaly.com/learn/machine-learning-foundations/slice-based-error-analysis"},"next":{"slug":"how-a-tree-chooses-a-split","title":"How a tree chooses where to cut","url":"https://addaly.com/learn/machine-learning-foundations/how-a-tree-chooses-a-split"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The classic algorithms, and when each is right', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/the-classic-algorithms (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The classic algorithms, and when each is right","courseTitle":"Machine Learning, Foundations","module":{"index":6,"slug":"trees-and-ensembles","title":"Trees, ensembles, and the algorithms that win on tables"},"answer":null,"keyIdea":"On tabular data, use linear models as a floor and gradient boosting as the answer; everything else needs a stated reason.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-short-catalogue","runs":[{"text":"A short catalogue"}]},{"t":"p","runs":[{"text":"Eight families cover almost everything built before deep learning, and most of what is still built on tables today. Each is a different answer to “what shape may the function take”."}]},{"t":"p","runs":[{"text":"Linear and logistic regression.","b":true},{"text":" Weighted sum of the inputs. Linear regression predicts a number; logistic regression squashes the sum into a probability. Small, fast, and each coefficient is a readable statement: this feature moves the log-odds by this much. Cannot represent interactions or curves unless you build them in as features. Use it as your floor (the score every other model must beat to justify itself) and as your answer whenever you must explain a decision to a regulator or a customer."}]},{"t":"p","runs":[{"text":"k-nearest neighbours.","b":true},{"text":" No training. To predict, find the k most similar rows you have seen and take their average or majority vote. Everything hinges on the distance function, so features must be scaled and irrelevant columns actively hurt. Falls apart in high dimensions, where everything is roughly equidistant from everything. Good when you have few examples, a distance that means something, and you want a baseline in ten minutes."}]},{"t":"p","runs":[{"text":"Decision trees.","b":true},{"text":" A learned sequence of yes/no questions. “Is income above ₹40,000? Then is tenure above 8 months?” Handles non-linearity and interactions for free, needs no scaling, and takes mixed data types. A single tree is readable enough to print and put in front of a domain expert, which is sometimes the entire reason to use one. On its own it is unstable (change 5% of the data and the tree can be different) and it overfits unless pruned."}]},{"t":"p","runs":[{"text":"Random forest.","b":true},{"text":" Hundreds of trees, each on a bootstrap sample of rows and a random subset of features, averaged. Averaging cancels the instability. Very hard to make it perform badly, robust to defaults, and the honest workhorse when you do not want to think about hyperparameters."}]},{"t":"p","runs":[{"text":"Gradient boosting","b":true},{"text":" (XGBoost, LightGBM, CatBoost). Trees again, but built in sequence, each one fitted to the errors the previous ones left behind. Consistently the strongest thing on tabular data and has been for a decade. The cost is hyperparameters that matter (learning rate, depth, number of trees, regularization) and a genuine ability to overfit if you set them carelessly."}]},{"t":"p","runs":[{"text":"Support vector machines.","b":true},{"text":" Find the boundary with the widest margin between classes; the kernel trick lets that boundary be curved. Excellent when you have many features and few rows, which is why they still appear in genomics and other wide-data fields. Training scales badly past tens of thousands of rows, and the output is not a probability without extra work."}]},{"t":"p","runs":[{"text":"Naive Bayes.","b":true},{"text":" Applies Bayes’ rule while pretending every feature is independent, which is false. It is fast, needs very little data, and works better than the assumption deserves for text. Its probabilities are badly calibrated: treat them as a ranking, never as a number."}]},{"t":"p","runs":[{"text":"k-means.","b":true},{"text":" Unsupervised. Partition rows into k groups by distance to k moving centres. Cheap and useful for exploration. You must choose k, it assumes roughly round and similarly sized clusters, and it returns clusters whether or not any exist."}]},{"t":"h2","id":"the-default","runs":[{"text":"The default"}]},{"t":"p","runs":[{"text":"For a table of rows and columns:"}]},{"t":"ol","start":1,"items":[[{"text":"Fit logistic or linear regression. Record the score. That is your floor."}],[{"text":"Fit gradient boosting with sensible defaults. That is usually your ceiling."}],[{"text":"If the gap between them is small, your features are the constraint. Go back to Lesson 3."}],[{"text":"Reach for anything else only for a stated reason: a printable tree because a committee must read it, an SVM because you have 300 rows and 8,000 columns, k-NN because a good distance is the whole domain."}]]},{"t":"p","runs":[{"text":"Deep learning does not belong in that list for tabular data. Repeated benchmarks find boosted trees ahead of neural networks on medium-sized tables, and the neural network costs more to tune and run. Use networks where they win: images, audio, text, and anything sequential."}]},{"t":"h2","id":"what-the-gap-between-two-models-tells-you","runs":[{"text":"What the gap between two models tells you"}]},{"t":"p","runs":[{"text":"This is the useful move, and it costs an afternoon."}]},{"t":"p","runs":[{"text":"Logistic regression can only draw a straight boundary. Gradient boosting can carve arbitrary regions and capture interactions between features. If you fit both and the scores are within a point of each other, that is evidence: there is little non-linear structure left for a flexible model to find, so the ceiling is set by what your columns contain."}]},{"t":"p","runs":[{"text":"The reflex at that point is to reach for something more powerful still. It is the wrong reflex. Two model families with very different expressive power agreeing on a number is a message about the data, and the answer is a better feature, a better label, or a column you do not yet have."}]}],"check":{"question":[{"text":"A team predicts crop yield from soil and weather columns. Their logistic-style linear model scores poorly. They switch to a random forest and the score moves by half a point. What does that most likely tell them?"}],"options":[{"text":[{"text":"The columns they have do not carry much information about yield."}],"correct":true,"tempting":null},{"text":[{"text":"Random forests are not strong enough; they should try gradient boosting or a neural network next."}],"correct":false,"tempting":[{"text":"There is a real ladder of model power, so a flat result reads naturally as not having climbed high enough yet."}]},{"text":[{"text":"The two models are broadly similar, so a similar score is expected and uninformative."}],"correct":false,"tempting":[{"text":"Both count as standard ML models, but a forest can express curves and interactions that a linear model structurally cannot, which is exactly why their agreement carries information."}]},{"text":[{"text":"They need more training rows."}],"correct":false,"tempting":[{"text":"More data reliably fixes variance problems, but here two model families with very different capacity land in the same place, which points at the inputs rather than the sample size."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"slice-based-error-analysis","title":"The average is hiding the failure","minutes":9},"next":{"slug":"how-a-tree-chooses-a-split","title":"How a tree chooses where to cut","minutes":9}}}