AutoML: what it automates, and what it cannot

What it does

AutoML systems search over preprocessing steps, model families, hyperparameters, and often ensembles of the results, using cross-validation to choose. You supply a table and a target column; it returns a fitted pipeline.

The free options are good and worth knowing by name:

• auto-sklearn: searches scikit-learn pipelines, uses meta-learning from previous datasets to start the search in a promising region.
• AutoGluon: from Amazon, free and open source. Trains several strong models and stacks them. It is frequently the strongest of the free options on tabular data with no tuning at all.
• FLAML: from Microsoft, free, optimised for finding a good model quickly on a small compute budget.
• H2O AutoML: free, mature, with a usable interface.
• TPOT: searches pipelines with genetic programming, and outputs readable Python code for the winner, which is useful for learning.

The paid ones (Vertex AI, DataRobot, SageMaker Autopilot) add hosting, monitoring and an interface. The modelling is not meaningfully better than AutoGluon.

from autogluon.tabular import TabularPredictor
p = TabularPredictor(label="target", eval_metric="roc_auc").fit(train_df)
p.leaderboard()

When it is the right call

As a baseline. Run it for an hour on any new problem. Now you know roughly what is achievable, and your hand-built model has a number to beat. If AutoML gets 0.86 and two weeks of your work gets 0.84, that is worth knowing on day one rather than at the end.

When nobody on the team is a specialist. A small business with a real prediction problem and no data scientist gets far more from AutoGluon than from a poorly tuned model built by someone learning as they go.

When time matters more than the last two points. Many business problems are worth solving to 90% of achievable quality this week.

As a check on your own work. If AutoML substantially beats your model, you have made a mistake somewhere and it is worth finding.

What it does not do

This is the part the marketing does not cover, and it maps exactly onto the modules of this course.

It does not frame the problem. What one row is, what the target means, when the prediction is made. Module 1. AutoML accepts whatever table you hand it, including the wrong one.

It does not build features. This is the largest source of accuracy on tabular problems, it comes from domain knowledge, and no search invents debt / income or days_since_last_purchase from a schema. AutoML tunes a model over the features you gave it.

It does not detect leakage. In fact it makes leakage worse, because a search that maximises validation score will enthusiastically select whatever configuration best exploits the leaked column. A leaked feature plus AutoML produces a spectacular number and a useless model.

It does not choose the metric or the threshold. It optimises what you specified. Specifying the wrong thing, at scale, is what AutoML is unusually good at.

It does not tell you whether the sample is representative. Module 3’s selection bias is invisible to any search.

It does not decide whether the model should be built. Module 1’s argument about rules, and module 5’s about fairness, are outside its scope entirely.

The honest summary: AutoML automates the part of this course that is easiest to automate, which is model selection and hyperparameter tuning, and that was never where most of the value was.

The costs

Compute. A serious search is hundreds of model fits. On a free tier this means hours.

Complexity of the result. A stacked ensemble of 12 models is slow to serve, large to store, and impossible to explain to anyone.

Overfitting the validation set. Hundreds of configurations evaluated against one validation set is exactly module 4’s problem in concentrated form. Keep a test set the search has never touched, and expect the final number to be below the search’s best.

Skill. A team that only runs AutoML does not develop the judgement to notice when the output is wrong, and the output being wrong is not rare.

How to actually use it

Use it as an instrument rather than a replacement. Run it early to establish what is achievable. Spend your time on framing, features and evaluation: the things it cannot do. Then run it again at the end on your engineered features, and ship whichever is better.

That use is unambiguously good practice, and it is not the use the products are sold for.