Tabular and time series: the honest answer

Gradient boosting still wins on tables

If your problem is predicting a number or a class from rows of structured columns (churn, credit risk, demand, conversion) the honest answer is that gradient-boosted trees remain the strongest default, and this has been tested repeatedly against deep learning on collections of real tabular datasets with consistent results.

The reasons are structural rather than incidental. Tabular features are heterogeneous and unordered, which suits axis-aligned splits and gives neural networks no useful inductive bias to exploit. Trees are robust to uninformative features, to outliers and to unscaled inputs. They train in seconds, need almost no tuning to be competitive, and tell you which features mattered.

XGBoost, LightGBM and CatBoost are all free. On a typical business table any of them will beat a fine-tuned language model on accuracy, by orders of magnitude on cost, and with an explanation of its predictions you can show somebody.

So: do not fine-tune a language model to predict a number from a table. That sentence saves more time than most of this course.

Where a language model does earn its place

Three specific jobs, all around the model rather than in place of it:

Free-text columns. A complaint field, a doctor’s note, a product description. Turn it into features (embeddings, or a small classifier’s output) and feed those into the tree model alongside the numeric columns. This is a genuine and frequently large gain, and it is the embedding and classifier work from earlier in this module.

Entity resolution and cleaning. Matching “Acme Ltd.” to “ACME Limited”, normalising inconsistent categories, parsing addresses. Language models are good at this and it is difficult to do any other way.

Writing the analysis. Generating the modelling code, explaining the result, drafting the report. Useful, and not a prediction task.

TabPFN, the interesting exception

One genuine exception is worth knowing: TabPFN is a transformer pretrained on synthetic tabular problems that performs classification by in-context learning: you pass the training rows in the prompt and it predicts, with no fitting step at all.

On small tables it is competitive with well-tuned boosted trees and takes seconds rather than a tuning session. Its constraint is size: it is designed for small datasets, up to roughly the low thousands of rows and a modest number of features, with later versions relaxing those limits. Above that, back to trees.

Time-series foundation models

A real development, and one worth being precise about. Chronos, TimesFM, Moirai and Lag-Llama are transformer models pretrained on very large collections of time series, and they forecast unseen series zero-shot, without being fitted to them.

They are useful in a specific situation: many series, each too short to fit a model to individually. Forecasting demand for 40,000 products where most have eighteen months of history is exactly the case, and it is a common one.

They can be fine-tuned on your own series, and this usually helps.

What to be careful about: the benchmark comparisons are contested, and the central concern is contamination. These models were pretrained on large public collections of time series, and the standard evaluation benchmarks are drawn from public time series. Whether a given result reflects forecasting ability or partial memorisation is a live methodological argument, and the field has not settled it.

The baseline that settles it

Whatever you use, benchmark against seasonal naive: predict that this week equals the same week last year, or last season.

yhat = y[-seasonal_period:]        # that is the whole model

It is one line, it costs nothing, and it beats a surprising share of deployed forecasting systems. If your foundation model cannot beat seasonal naive plus a simple statistical method such as exponential smoothing or ARIMA (both free, both in statsmodels) on your series, you have learned something important for the price of ten minutes.

That is the discipline this whole course keeps returning to. The impressive method has to beat the dull one on your data, and the dull one has to actually be run for the comparison to mean anything.