Searching hyperparameters without fooling yourself

Parameters and hyperparameters

Parameters are learned from the data: the coefficients, the split thresholds, the network weights. Hyperparameters are set by you before training: max_depth, alpha, learning_rate, n_estimators.

The distinction matters because parameters are fitted on the training set and hyperparameters must be chosen on the validation set. Choosing them on the training set means picking whatever fits it hardest, which is always the most complex option available.

Grid search, and why it wastes most of its budget

from sklearn.model_selection import GridSearchCV
grid = {"model__max_depth": [3, 5, 7],
        "model__learning_rate": [0.01, 0.05, 0.1],
        "model__min_samples_leaf": [10, 20, 50]}
search = GridSearchCV(pipe, grid, cv=5, scoring="roc_auc", n_jobs=-1)

Every combination: 3 x 3 x 3 = 27 configurations, times 5 folds, is 135 fits. Adding one more value to one parameter takes it to 180.

The deeper problem is not cost. It is that in most problems only two or three hyperparameters matter, and grid search spends the same budget on the ones that do not. With 27 configurations, the important parameter was tried at exactly three values.

Random search, and why it wins

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, randint
dist = {"model__max_depth": randint(2, 12),
        "model__learning_rate": loguniform(1e-3, 3e-1),
        "model__min_samples_leaf": randint(5, 100)}
search = RandomizedSearchCV(pipe, dist, n_iter=60, cv=5, n_jobs=-1,
                            random_state=0)

Sixty random draws from those distributions try sixty distinct values of each parameter, rather than three. When one parameter dominates, and one usually does, random search explores it sixty times more finely for the same budget. Bergstra and Bengio’s 2012 result made this argument, and it has held up: for the same number of fits, random search generally finds a better configuration than grid search.

Use loguniform for anything whose useful range spans orders of magnitude: learning rates, regularisation strengths, C. Sampling alpha uniformly between 0.001 and 100 puts 99.9% of your draws above 0.1, which is almost certainly the wrong end.

Bayesian and successive halving

Bayesian optimisation builds a model of the score as a function of the hyperparameters and picks the next point to try where the expected improvement is greatest. optuna is the free standard here and it is good. It pays off when each fit is expensive; when a fit takes two seconds, the overhead of being clever exceeds the savings.

Successive halving starts many configurations on a small subset of the data, kills the worst half, doubles the resources for the survivors, and repeats. HalvingRandomSearchCV is in scikit-learn. It is a large speedup and it carries an assumption worth knowing: that a configuration which looks poor on 10% of the data would also be poor on 100%. That assumption is usually but not always right, and it fails most often for configurations that need more data to show their value.

Which hyperparameters actually matter

A short list, because the space is large and most of it is irrelevant.

• Gradient boosting: learning_rate and n_estimators together (halve one, double the other), then max_depth or num_leaves, then min_child_weight and the subsampling fractions. Everything else is fine-tuning.
• Random forest: max_features first, then min_samples_leaf. n_estimators is not really a hyperparameter: more is never worse, only slower, so set it as high as you can afford and stop tuning it.
• Linear models: C or alpha, on a log scale. That is essentially the whole search.
• Neural networks: learning rate, by a wide margin, then batch size and weight decay.

Tune three parameters well rather than nine badly.

The cost you are accumulating

Every configuration you evaluate is another look at the validation data, and the more you look, the more the best score reflects the particular noise in that set. Two hundred configurations is two hundred chances for something to score well by luck.

This is the subject of the next lesson, and it is the reason for the discipline that follows: tune on cross-validation over the training data, and keep a test set that no search has ever seen. When you finally score on it and get a number a point or two below your best cross-validated score, that is not a disappointment: it is the search bias becoming visible, which is exactly what the test set is for.