AutoML: what it genuinely does, and what it cannot do for you

What it is

Point a tool at a spreadsheet, say which column you want predicted, and it tries a range of methods, tunes them, and hands back the best with an accuracy figure. Paid versions exist on every cloud platform; free ones do the same job on a laptop.

This is a real convenience. Choosing between a dozen algorithms and tuning their settings used to take a specialist days, and it is now a button. For a small business it means you can get a competent model without learning what a gradient-boosted tree is.

What it automates

Model selection. Hyperparameter tuning. Some encoding of categories, some handling of missing values. A cross-validation score.

That list is worth reading twice, because of what is not on it.

What it cannot do, and hands back to you unchanged

The question. It cannot tell you whether predicting lapse is the useful question, or whether you should have been predicting which customers respond to a voucher, which is a different and usually more actionable thing.

The definition. Someone has to decide what counts as lapsed. AutoML predicts whatever column you point at, including a badly defined one, with great efficiency.

The columns. As the previous lessons said, features beat algorithms. AutoML does not know that a lengthening gap between orders matters; you have to compute that column and give it.

The leak. This is the important one. AutoML will happily build a superb model on a leaking column and report 97%. It has no way to know that deposit_paid_date did not exist at prediction time, because that is a fact about your business, not about the data.

The threshold. It reports a score at some default. The cut-off that matches what an intervention costs you is yours to set.

Whether to act. A model with 30% precision might be worth acting on or might not, depending entirely on what a wrong flag costs. No tool computes that.

The free path, which is adequate

PyCaret: a few lines of Python, compares a dozen models automatically, free and open source. Orange: visual, drag-and-drop, no code, free. Good for learning what the methods actually do because you can see the pipeline. KNIME: free desktop version, visual workflows, widely used in industry. scikit-learn: not AutoML, but its RandomForestClassifier with default settings is a strong baseline in about four lines, and beating it meaningfully is harder than people expect.

Paid AutoML on the big clouds adds scale, deployment and monitoring. At small-business volumes you are paying for infrastructure you will not use.

Read the score it gives you, carefully

AutoML reports a cross-validated figure, which means it split your data repeatedly and averaged the results. Two things to know about that number.

It is computed on the data you supplied, which is the past. It is an estimate of how the model would have done on history, and it assumes next year resembles last year. When a competitor opens or a product line changes, that assumption stops holding and the score does not update.

And by default most tools split randomly, which is the time leak from the previous lesson. Look for the option to split by date, and use it for anything with a time dimension. In several popular tools it is not the default, and the difference in reported accuracy between the two settings is often larger than the difference between any two algorithms the tool compared.

The comparison that keeps you honest

Whatever AutoML produces, compare it against two things.

The rule: sort by days since last order, take the top hundred. Measure its precision and recall on the same holdout.

The simplest model: logistic regression with five sensible columns. It also tells you, in plain numbers, which factors mattered and in which direction, which a boosted ensemble will not.

In small-business data, the gap between logistic regression and the best AutoML result is frequently a couple of percentage points. If that is the gap, take the simple model: you can explain it to your staff, debug it when it drifts, and see immediately when one of its inputs stops making sense.

When to use it

When you have tens of thousands of rows, a defined target, features you built deliberately, and a rule-based baseline you want to beat. Then AutoML is a sensible hour’s work.

When you have four hundred rows and a hopeful question, it will still produce a model with a confident score, and the next lesson is about why that number is worse than nothing.