{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"automl-and-when-it-is-right","title":"AutoML: what it automates, and what it cannot","url":"https://addaly.com/learn/machine-learning-foundations/automl-and-when-it-is-right","position":89,"of":92,"minutes":8,"answer":"AutoML automates model selection and hyperparameter search, genuinely well, and free, but it cannot frame the problem, build features, detect leakage or choose the metric, which is where most of a model's quality is determined.","keyIdea":"AutoML automates model selection and hyperparameter search, genuinely well, and free, but it cannot frame the problem, build features, detect leakage or choose the metric, which is where most of a model's quality is determined."},"body":{"format":"markdown","text":"## What it does\n\nAutoML systems search over preprocessing steps, model families, hyperparameters, and often ensembles of the results, using cross-validation to choose. You supply a table and a target column; it returns a fitted pipeline.\n\nThe free options are good and worth knowing by name:\n\n- **auto-sklearn**: searches scikit-learn pipelines, uses meta-learning from previous datasets to start the search in a promising region.\n- **AutoGluon**: from Amazon, free and open source. Trains several strong models and stacks them. It is frequently the strongest of the free options on tabular data with no tuning at all.\n- **FLAML**: from Microsoft, free, optimised for finding a good model quickly on a small compute budget.\n- **H2O AutoML**: free, mature, with a usable interface.\n- **TPOT**: searches pipelines with genetic programming, and outputs readable Python code for the winner, which is useful for learning.\n\nThe paid ones (Vertex AI, DataRobot, SageMaker Autopilot) add hosting, monitoring and an interface. The modelling is not meaningfully better than AutoGluon.\n\n```python\nfrom autogluon.tabular import TabularPredictor\np = TabularPredictor(label=\"target\", eval_metric=\"roc_auc\").fit(train_df)\np.leaderboard()\n```\n\n## When it is the right call\n\n**As a baseline.** Run it for an hour on any new problem. Now you know roughly what is achievable, and your hand-built model has a number to beat. If AutoML gets 0.86 and two weeks of your work gets 0.84, that is worth knowing on day one rather than at the end.\n\n**When nobody on the team is a specialist.** A small business with a real prediction problem and no data scientist gets far more from AutoGluon than from a poorly tuned model built by someone learning as they go.\n\n**When time matters more than the last two points.** Many business problems are worth solving to 90% of achievable quality this week.\n\n**As a check on your own work.** If AutoML substantially beats your model, you have made a mistake somewhere and it is worth finding.\n\n## What it does not do\n\nThis is the part the marketing does not cover, and it maps exactly onto the modules of this course.\n\n**It does not frame the problem.** What one row is, what the target means, when the prediction is made. Module 1. AutoML accepts whatever table you hand it, including the wrong one.\n\n**It does not build features.** This is the largest source of accuracy on tabular problems, it comes from domain knowledge, and no search invents `debt / income` or `days_since_last_purchase` from a schema. AutoML tunes a model over the features you gave it.\n\n**It does not detect leakage.** In fact it makes leakage worse, because a search that maximises validation score will enthusiastically select whatever configuration best exploits the leaked column. A leaked feature plus AutoML produces a spectacular number and a useless model.\n\n**It does not choose the metric or the threshold.** It optimises what you specified. Specifying the wrong thing, at scale, is what AutoML is unusually good at.\n\n**It does not tell you whether the sample is representative.** Module 3's selection bias is invisible to any search.\n\n**It does not decide whether the model should be built.** Module 1's argument about rules, and module 5's about fairness, are outside its scope entirely.\n\nThe honest summary: **AutoML automates the part of this course that is easiest to automate, which is model selection and hyperparameter tuning, and that was never where most of the value was.**\n\n## The costs\n\n**Compute.** A serious search is hundreds of model fits. On a free tier this means hours.\n\n**Complexity of the result.** A stacked ensemble of 12 models is slow to serve, large to store, and impossible to explain to anyone.\n\n**Overfitting the validation set.** Hundreds of configurations evaluated against one validation set is exactly module 4's problem in concentrated form. Keep a test set the search has never touched, and expect the final number to be below the search's best.\n\n**Skill.** A team that only runs AutoML does not develop the judgement to notice when the output is wrong, and the output being wrong is not rare.\n\n## How to actually use it\n\nUse it as an instrument rather than a replacement. Run it early to establish what is achievable. Spend your time on framing, features and evaluation: the things it cannot do. Then run it again at the end on your engineered features, and ship whichever is better.\n\nThat use is unambiguously good practice, and it is not the use the products are sold for."},"check":{"question":"A team runs AutoML on a churn dataset and gets 0.97 AUC, far above their hand-built model's 0.82. What should they do first?","options":[{"text":"Deploy it, since the search evaluated hundreds of configurations against proper cross-validation.","correct":false,"tempting":"Cross-validation inside the search does not protect against a leaked column, which contaminates every fold equally."},{"text":"Investigate for leakage, because a large unexplained jump on a hard problem is the signature of a feature containing the answer, and a search that maximises validation score selects the configuration that exploits it best.","correct":true},{"text":"Rerun with a larger time budget, since the search may not have converged and the true score could be higher still.","correct":false,"tempting":"Extending the search compounds the problem rather than examining it; the anomaly is the size of the gain, not insufficient search."},{"text":"Retrain their hand-built model with the hyperparameters AutoML selected, to close the gap.","correct":false,"tempting":"Copying hyperparameters would transfer the same exploitation of a leaked feature into the hand-built model without ever diagnosing it."}]},"prev":{"slug":"reproducibility","title":"Being able to build the same model twice","url":"https://addaly.com/learn/machine-learning-foundations/reproducibility"},"next":{"slug":"making-a-model-smaller","title":"Making a model small enough to be affordable","url":"https://addaly.com/learn/machine-learning-foundations/making-a-model-smaller"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'AutoML: what it automates, and what it cannot', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/automl-and-when-it-is-right (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"AutoML: what it automates, and what it cannot","courseTitle":"Machine Learning, Foundations","module":{"index":9,"slug":"into-the-world","title":"Into the world, and keeping it honest"},"answer":null,"keyIdea":"AutoML automates model selection and hyperparameter search, genuinely well, and free, but it cannot frame the problem, build features, detect leakage or choose the metric, which is where most of a model’s quality is determined.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-it-does","runs":[{"text":"What it does"}]},{"t":"p","runs":[{"text":"AutoML systems search over preprocessing steps, model families, hyperparameters, and often ensembles of the results, using cross-validation to choose. You supply a table and a target column; it returns a fitted pipeline."}]},{"t":"p","runs":[{"text":"The free options are good and worth knowing by name:"}]},{"t":"ul","items":[[{"text":"auto-sklearn","b":true},{"text":": searches scikit-learn pipelines, uses meta-learning from previous datasets to start the search in a promising region."}],[{"text":"AutoGluon","b":true},{"text":": from Amazon, free and open source. Trains several strong models and stacks them. It is frequently the strongest of the free options on tabular data with no tuning at all."}],[{"text":"FLAML","b":true},{"text":": from Microsoft, free, optimised for finding a good model quickly on a small compute budget."}],[{"text":"H2O AutoML","b":true},{"text":": free, mature, with a usable interface."}],[{"text":"TPOT","b":true},{"text":": searches pipelines with genetic programming, and outputs readable Python code for the winner, which is useful for learning."}]]},{"t":"p","runs":[{"text":"The paid ones (Vertex AI, DataRobot, SageMaker Autopilot) add hosting, monitoring and an interface. The modelling is not meaningfully better than AutoGluon."}]},{"t":"code","lang":"python","label":"Python","text":"from autogluon.tabular import TabularPredictor\np = TabularPredictor(label=\"target\", eval_metric=\"roc_auc\").fit(train_df)\np.leaderboard()"},{"t":"h2","id":"when-it-is-the-right-call","runs":[{"text":"When it is the right call"}]},{"t":"p","runs":[{"text":"As a baseline.","b":true},{"text":" Run it for an hour on any new problem. Now you know roughly what is achievable, and your hand-built model has a number to beat. If AutoML gets 0.86 and two weeks of your work gets 0.84, that is worth knowing on day one rather than at the end."}]},{"t":"p","runs":[{"text":"When nobody on the team is a specialist.","b":true},{"text":" A small business with a real prediction problem and no data scientist gets far more from AutoGluon than from a poorly tuned model built by someone learning as they go."}]},{"t":"p","runs":[{"text":"When time matters more than the last two points.","b":true},{"text":" Many business problems are worth solving to 90% of achievable quality this week."}]},{"t":"p","runs":[{"text":"As a check on your own work.","b":true},{"text":" If AutoML substantially beats your model, you have made a mistake somewhere and it is worth finding."}]},{"t":"h2","id":"what-it-does-not-do","runs":[{"text":"What it does not do"}]},{"t":"p","runs":[{"text":"This is the part the marketing does not cover, and it maps exactly onto the modules of this course."}]},{"t":"p","runs":[{"text":"It does not frame the problem.","b":true},{"text":" What one row is, what the target means, when the prediction is made. Module 1. AutoML accepts whatever table you hand it, including the wrong one."}]},{"t":"p","runs":[{"text":"It does not build features.","b":true},{"text":" This is the largest source of accuracy on tabular problems, it comes from domain knowledge, and no search invents "},{"text":"debt / income","code":true},{"text":" or "},{"text":"days_since_last_purchase","code":true},{"text":" from a schema. AutoML tunes a model over the features you gave it."}]},{"t":"p","runs":[{"text":"It does not detect leakage.","b":true},{"text":" In fact it makes leakage worse, because a search that maximises validation score will enthusiastically select whatever configuration best exploits the leaked column. A leaked feature plus AutoML produces a spectacular number and a useless model."}]},{"t":"p","runs":[{"text":"It does not choose the metric or the threshold.","b":true},{"text":" It optimises what you specified. Specifying the wrong thing, at scale, is what AutoML is unusually good at."}]},{"t":"p","runs":[{"text":"It does not tell you whether the sample is representative.","b":true},{"text":" Module 3’s selection bias is invisible to any search."}]},{"t":"p","runs":[{"text":"It does not decide whether the model should be built.","b":true},{"text":" Module 1’s argument about rules, and module 5’s about fairness, are outside its scope entirely."}]},{"t":"p","runs":[{"text":"The honest summary: "},{"text":"AutoML automates the part of this course that is easiest to automate, which is model selection and hyperparameter tuning, and that was never where most of the value was.","b":true}]},{"t":"h2","id":"the-costs","runs":[{"text":"The costs"}]},{"t":"p","runs":[{"text":"Compute.","b":true},{"text":" A serious search is hundreds of model fits. On a free tier this means hours."}]},{"t":"p","runs":[{"text":"Complexity of the result.","b":true},{"text":" A stacked ensemble of 12 models is slow to serve, large to store, and impossible to explain to anyone."}]},{"t":"p","runs":[{"text":"Overfitting the validation set.","b":true},{"text":" Hundreds of configurations evaluated against one validation set is exactly module 4’s problem in concentrated form. Keep a test set the search has never touched, and expect the final number to be below the search’s best."}]},{"t":"p","runs":[{"text":"Skill.","b":true},{"text":" A team that only runs AutoML does not develop the judgement to notice when the output is wrong, and the output being wrong is not rare."}]},{"t":"h2","id":"how-to-actually-use-it","runs":[{"text":"How to actually use it"}]},{"t":"p","runs":[{"text":"Use it as an instrument rather than a replacement. Run it early to establish what is achievable. Spend your time on framing, features and evaluation: the things it cannot do. Then run it again at the end on your engineered features, and ship whichever is better."}]},{"t":"p","runs":[{"text":"That use is unambiguously good practice, and it is not the use the products are sold for."}]}],"check":{"question":[{"text":"A team runs AutoML on a churn dataset and gets 0.97 AUC, far above their hand-built model’s 0.82. What should they do first?"}],"options":[{"text":[{"text":"Deploy it, since the search evaluated hundreds of configurations against proper cross-validation."}],"correct":false,"tempting":[{"text":"Cross-validation inside the search does not protect against a leaked column, which contaminates every fold equally."}]},{"text":[{"text":"Investigate for leakage, because a large unexplained jump on a hard problem is the signature of a feature containing the answer, and a search that maximises validation score selects the configuration that exploits it best."}],"correct":true,"tempting":null},{"text":[{"text":"Rerun with a larger time budget, since the search may not have converged and the true score could be higher still."}],"correct":false,"tempting":[{"text":"Extending the search compounds the problem rather than examining it; the anomaly is the size of the gain, not insufficient search."}]},{"text":[{"text":"Retrain their hand-built model with the hyperparameters AutoML selected, to close the gap."}],"correct":false,"tempting":[{"text":"Copying hyperparameters would transfer the same exploitation of a leaked feature into the hand-built model without ever diagnosing it."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"reproducibility","title":"Being able to build the same model twice","minutes":8},"next":{"slug":"making-a-model-smaller","title":"Making a model small enough to be affordable","minutes":8}}}