{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"hyperparameter-search","title":"Searching hyperparameters without fooling yourself","url":"https://addaly.com/learn/machine-learning-foundations/hyperparameter-search","position":38,"of":92,"minutes":9,"answer":"Random search over log-scaled distributions beats grid search at equal cost because it tries many distinct values of the parameters that matter.","keyIdea":"Random search over log-scaled distributions beats grid search at equal cost because it tries many distinct values of the parameters that matter, and every configuration evaluated adds selection bias that only an untouched test set can measure."},"body":{"format":"markdown","text":"## Parameters and hyperparameters\n\nParameters are learned from the data: the coefficients, the split thresholds, the network weights. Hyperparameters are set by you before training: `max_depth`, `alpha`, `learning_rate`, `n_estimators`.\n\nThe distinction matters because parameters are fitted on the training set and hyperparameters must be chosen on the validation set. Choosing them on the training set means picking whatever fits it hardest, which is always the most complex option available.\n\n## Grid search, and why it wastes most of its budget\n\n```python\nfrom sklearn.model_selection import GridSearchCV\ngrid = {\"model__max_depth\": [3, 5, 7],\n        \"model__learning_rate\": [0.01, 0.05, 0.1],\n        \"model__min_samples_leaf\": [10, 20, 50]}\nsearch = GridSearchCV(pipe, grid, cv=5, scoring=\"roc_auc\", n_jobs=-1)\n```\n\nEvery combination: 3 x 3 x 3 = 27 configurations, times 5 folds, is 135 fits. Adding one more value to one parameter takes it to 180.\n\nThe deeper problem is not cost. It is that in most problems only two or three hyperparameters matter, and grid search spends the same budget on the ones that do not. With 27 configurations, the important parameter was tried at exactly three values.\n\n## Random search, and why it wins\n\n```python\nfrom sklearn.model_selection import RandomizedSearchCV\nfrom scipy.stats import loguniform, randint\ndist = {\"model__max_depth\": randint(2, 12),\n        \"model__learning_rate\": loguniform(1e-3, 3e-1),\n        \"model__min_samples_leaf\": randint(5, 100)}\nsearch = RandomizedSearchCV(pipe, dist, n_iter=60, cv=5, n_jobs=-1,\n                            random_state=0)\n```\n\nSixty random draws from those distributions try sixty *distinct values* of each parameter, rather than three. When one parameter dominates, and one usually does, random search explores it sixty times more finely for the same budget. Bergstra and Bengio's 2012 result made this argument, and it has held up: for the same number of fits, random search generally finds a better configuration than grid search.\n\nUse `loguniform` for anything whose useful range spans orders of magnitude: learning rates, regularisation strengths, `C`. Sampling `alpha` uniformly between 0.001 and 100 puts 99.9% of your draws above 0.1, which is almost certainly the wrong end.\n\n## Bayesian and successive halving\n\n**Bayesian optimisation** builds a model of the score as a function of the hyperparameters and picks the next point to try where the expected improvement is greatest. `optuna` is the free standard here and it is good. It pays off when each fit is expensive; when a fit takes two seconds, the overhead of being clever exceeds the savings.\n\n**Successive halving** starts many configurations on a small subset of the data, kills the worst half, doubles the resources for the survivors, and repeats. `HalvingRandomSearchCV` is in scikit-learn. It is a large speedup and it carries an assumption worth knowing: that a configuration which looks poor on 10% of the data would also be poor on 100%. That assumption is usually but not always right, and it fails most often for configurations that need more data to show their value.\n\n## Which hyperparameters actually matter\n\nA short list, because the space is large and most of it is irrelevant.\n\n- **Gradient boosting**: `learning_rate` and `n_estimators` together (halve one, double the other), then `max_depth` or `num_leaves`, then `min_child_weight` and the subsampling fractions. Everything else is fine-tuning.\n- **Random forest**: `max_features` first, then `min_samples_leaf`. `n_estimators` is not really a hyperparameter: more is never worse, only slower, so set it as high as you can afford and stop tuning it.\n- **Linear models**: `C` or `alpha`, on a log scale. That is essentially the whole search.\n- **Neural networks**: learning rate, by a wide margin, then batch size and weight decay.\n\nTune three parameters well rather than nine badly.\n\n## The cost you are accumulating\n\nEvery configuration you evaluate is another look at the validation data, and the more you look, the more the best score reflects the particular noise in that set. Two hundred configurations is two hundred chances for something to score well by luck.\n\nThis is the subject of the next lesson, and it is the reason for the discipline that follows: tune on cross-validation over the training data, and keep a test set that no search has ever seen. When you finally score on it and get a number a point or two below your best cross-validated score, that is not a disappointment: it is the search bias becoming visible, which is exactly what the test set is for."},"check":{"question":"A team searches ridge alpha over a uniform grid of 20 values between 0.01 and 200 and finds the best at 0.01, the smallest value tried. What does this suggest about the search design?","options":[{"text":"The penalty is unnecessary and they should refit with no regularisation at all.","correct":false,"tempting":"The optimum sitting at the boundary suggests looking further in that direction, but jumping to zero penalty skips the region between 0 and 0.01 where the answer may lie."},{"text":"Twenty values is too few, and a denser grid over the same range would find a better optimum.","correct":false,"tempting":"Density within the same range does not help when the best value is at the edge; the range itself is the problem."},{"text":"The optimum is at the lower boundary, and a uniform grid puts almost all its points above 10, so the informative region below 1 was barely sampled; use a log scale extending lower.","correct":true},{"text":"Cross-validation noise made the smallest alpha appear best by chance, and repeating with a different seed would give a different winner.","correct":false,"tempting":"Noise can shuffle nearby candidates, but a systematic result at the boundary usually indicates the range is wrong rather than that the estimate is unstable."}]},"prev":{"slug":"learning-curves-and-more-data","title":"Would more data help? The curve that answers it","url":"https://addaly.com/learn/machine-learning-foundations/learning-curves-and-more-data"},"next":{"slug":"the-validation-set-wears-out","title":"A validation set you have used two hundred times","url":"https://addaly.com/learn/machine-learning-foundations/the-validation-set-wears-out"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Searching hyperparameters without fooling yourself', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/hyperparameter-search (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Searching hyperparameters without fooling yourself","courseTitle":"Machine Learning, Foundations","module":{"index":4,"slug":"generalisation","title":"Generalisation, and the discipline of held-out data"},"answer":null,"keyIdea":"Random search over log-scaled distributions beats grid search at equal cost because it tries many distinct values of the parameters that matter, and every configuration evaluated adds selection bias that only an untouched test set can measure.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"parameters-and-hyperparameters","runs":[{"text":"Parameters and hyperparameters"}]},{"t":"p","runs":[{"text":"Parameters are learned from the data: the coefficients, the split thresholds, the network weights. Hyperparameters are set by you before training: "},{"text":"max_depth","code":true},{"text":", "},{"text":"alpha","code":true},{"text":", "},{"text":"learning_rate","code":true},{"text":", "},{"text":"n_estimators","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"The distinction matters because parameters are fitted on the training set and hyperparameters must be chosen on the validation set. Choosing them on the training set means picking whatever fits it hardest, which is always the most complex option available."}]},{"t":"h2","id":"grid-search-and-why-it-wastes-most-of-its-budget","runs":[{"text":"Grid search, and why it wastes most of its budget"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import GridSearchCV\ngrid = {\"model__max_depth\": [3, 5, 7],\n        \"model__learning_rate\": [0.01, 0.05, 0.1],\n        \"model__min_samples_leaf\": [10, 20, 50]}\nsearch = GridSearchCV(pipe, grid, cv=5, scoring=\"roc_auc\", n_jobs=-1)"},{"t":"p","runs":[{"text":"Every combination: 3 x 3 x 3 = 27 configurations, times 5 folds, is 135 fits. Adding one more value to one parameter takes it to 180."}]},{"t":"p","runs":[{"text":"The deeper problem is not cost. It is that in most problems only two or three hyperparameters matter, and grid search spends the same budget on the ones that do not. With 27 configurations, the important parameter was tried at exactly three values."}]},{"t":"h2","id":"random-search-and-why-it-wins","runs":[{"text":"Random search, and why it wins"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import RandomizedSearchCV\nfrom scipy.stats import loguniform, randint\ndist = {\"model__max_depth\": randint(2, 12),\n        \"model__learning_rate\": loguniform(1e-3, 3e-1),\n        \"model__min_samples_leaf\": randint(5, 100)}\nsearch = RandomizedSearchCV(pipe, dist, n_iter=60, cv=5, n_jobs=-1,\n                            random_state=0)"},{"t":"p","runs":[{"text":"Sixty random draws from those distributions try sixty "},{"text":"distinct values","i":true},{"text":" of each parameter, rather than three. When one parameter dominates, and one usually does, random search explores it sixty times more finely for the same budget. Bergstra and Bengio’s 2012 result made this argument, and it has held up: for the same number of fits, random search generally finds a better configuration than grid search."}]},{"t":"p","runs":[{"text":"Use "},{"text":"loguniform","code":true},{"text":" for anything whose useful range spans orders of magnitude: learning rates, regularisation strengths, "},{"text":"C","code":true},{"text":". Sampling "},{"text":"alpha","code":true},{"text":" uniformly between 0.001 and 100 puts 99.9% of your draws above 0.1, which is almost certainly the wrong end."}]},{"t":"h2","id":"bayesian-and-successive-halving","runs":[{"text":"Bayesian and successive halving"}]},{"t":"p","runs":[{"text":"Bayesian optimisation","b":true},{"text":" builds a model of the score as a function of the hyperparameters and picks the next point to try where the expected improvement is greatest. "},{"text":"optuna","code":true},{"text":" is the free standard here and it is good. It pays off when each fit is expensive; when a fit takes two seconds, the overhead of being clever exceeds the savings."}]},{"t":"p","runs":[{"text":"Successive halving","b":true},{"text":" starts many configurations on a small subset of the data, kills the worst half, doubles the resources for the survivors, and repeats. "},{"text":"HalvingRandomSearchCV","code":true},{"text":" is in scikit-learn. It is a large speedup and it carries an assumption worth knowing: that a configuration which looks poor on 10% of the data would also be poor on 100%. That assumption is usually but not always right, and it fails most often for configurations that need more data to show their value."}]},{"t":"h2","id":"which-hyperparameters-actually-matter","runs":[{"text":"Which hyperparameters actually matter"}]},{"t":"p","runs":[{"text":"A short list, because the space is large and most of it is irrelevant."}]},{"t":"ul","items":[[{"text":"Gradient boosting","b":true},{"text":": "},{"text":"learning_rate","code":true},{"text":" and "},{"text":"n_estimators","code":true},{"text":" together (halve one, double the other), then "},{"text":"max_depth","code":true},{"text":" or "},{"text":"num_leaves","code":true},{"text":", then "},{"text":"min_child_weight","code":true},{"text":" and the subsampling fractions. Everything else is fine-tuning."}],[{"text":"Random forest","b":true},{"text":": "},{"text":"max_features","code":true},{"text":" first, then "},{"text":"min_samples_leaf","code":true},{"text":". "},{"text":"n_estimators","code":true},{"text":" is not really a hyperparameter: more is never worse, only slower, so set it as high as you can afford and stop tuning it."}],[{"text":"Linear models","b":true},{"text":": "},{"text":"C","code":true},{"text":" or "},{"text":"alpha","code":true},{"text":", on a log scale. That is essentially the whole search."}],[{"text":"Neural networks","b":true},{"text":": learning rate, by a wide margin, then batch size and weight decay."}]]},{"t":"p","runs":[{"text":"Tune three parameters well rather than nine badly."}]},{"t":"h2","id":"the-cost-you-are-accumulating","runs":[{"text":"The cost you are accumulating"}]},{"t":"p","runs":[{"text":"Every configuration you evaluate is another look at the validation data, and the more you look, the more the best score reflects the particular noise in that set. Two hundred configurations is two hundred chances for something to score well by luck."}]},{"t":"p","runs":[{"text":"This is the subject of the next lesson, and it is the reason for the discipline that follows: tune on cross-validation over the training data, and keep a test set that no search has ever seen. When you finally score on it and get a number a point or two below your best cross-validated score, that is not a disappointment: it is the search bias becoming visible, which is exactly what the test set is for."}]}],"check":{"question":[{"text":"A team searches ridge alpha over a uniform grid of 20 values between 0.01 and 200 and finds the best at 0.01, the smallest value tried. What does this suggest about the search design?"}],"options":[{"text":[{"text":"The penalty is unnecessary and they should refit with no regularisation at all."}],"correct":false,"tempting":[{"text":"The optimum sitting at the boundary suggests looking further in that direction, but jumping to zero penalty skips the region between 0 and 0.01 where the answer may lie."}]},{"text":[{"text":"Twenty values is too few, and a denser grid over the same range would find a better optimum."}],"correct":false,"tempting":[{"text":"Density within the same range does not help when the best value is at the edge; the range itself is the problem."}]},{"text":[{"text":"The optimum is at the lower boundary, and a uniform grid puts almost all its points above 10, so the informative region below 1 was barely sampled; use a log scale extending lower."}],"correct":true,"tempting":null},{"text":[{"text":"Cross-validation noise made the smallest alpha appear best by chance, and repeating with a different seed would give a different winner."}],"correct":false,"tempting":[{"text":"Noise can shuffle nearby candidates, but a systematic result at the boundary usually indicates the range is wrong rather than that the estimate is unstable."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"learning-curves-and-more-data","title":"Would more data help? The curve that answers it","minutes":8},"next":{"slug":"the-validation-set-wears-out","title":"A validation set you have used two hundred times","minutes":8}}}