Hyperparameter

Definition

A hyperparameter is a setting you choose before training starts and that gradient descent never changes — the learning rate, the batch size, how many layers the network has. It lives outside the thing being optimized. Training spends its whole run adjusting the model's millions or billions of parameters to fit the data, but it treats every hyperparameter as a fixed constant, so a badly chosen one is not corrected by training more. It caps how good the run can get, and no amount of extra epochs lifts the ceiling.

The learning rate is the sharpest example, and the reason this term is worth understanding before any other tuning knob. Set it too high and the loss does not just train poorly — it blows up to NaN within a handful of steps, a failure most beginners mistake for a bug in their code. Set it too low and nothing is wrong except time: a run that should converge in an afternoon crawls for a week to reach the same place. One number, chosen by a human, decides whether iteration works at all.

How It Works

The line gradient descent will not cross

Training a model means minimizing a loss function — a single number measuring how wrong the model's outputs are. Gradient descent does this by computing, for every parameter, a gradient: the direction and amount that nudging that one number would reduce the loss. Multiply the gradient by a step size, subtract it, repeat. Over billions of steps the weights settle into values that fit the data. That is the inside of the training loop, and every number it touches is a learned parameter.

A hyperparameter is a knob on the outside of that loop. There is no gradient for it. You cannot differentiate the loss "with respect to the number of layers" partway through a run, because the number of layers is what defines which parameters exist in the first place — change it and you are training a different network, not nudging the current one. The same is true of the learning rate: gradient descent uses it to scale its steps, but it has no mechanism to adjust it, because it is the thing doing the adjusting. This is the whole distinction in one line: parameters are learned, hyperparameters are chosen. GPT-3 shipped with 175 billion learned parameters (Brown et al., 2020) and a few dozen chosen hyperparameters. The 175 billion are what training produced; the few dozen are what the researchers typed in before it started.

The learning rate, on a loss you can work out by hand

Real loss functions have no closed form, so it is hard to see what the learning rate does. Shrink the problem to a loss with exactly one parameter, L(θ) = θ², a simple upward parabola whose lowest point is at θ = 0. Its gradient is dL/dθ = 2θ, so one gradient-descent step with learning rate η is:

θ_next = θ − η · 2θ = θ · (1 − 2η)

Every step just multiplies θ by the fixed factor (1 − 2η). That single factor decides the entire fate of the run. Start at θ = 1 and watch what each learning rate does:

Three regimes fall out of one multiplication. When 0 < η < 1 the factor has magnitude below one and θ shrinks toward the minimum; too small and it shrinks so slowly the run is uselessly slow — at η = 0.001 it takes about 2,300 steps to get where η = 0.5 gets in one. At exactly η = 1 the factor is −1 and θ bounces between 1 and −1 forever, making progress on neither. Past η = 1 the factor exceeds one in magnitude and every step makes θ bigger: the loss doubles, then quadruples, then overflows. Nothing about the model failed; the step size was wrong.

The exact sweet spot here, η = 0.5, exists only because this toy loss is a clean parabola with a known curvature. Real loss surfaces have no such closed form, their curvature differs in every direction and changes as training proceeds — which is precisely why the learning rate has no safe default and why practitioners reach for schedules (a warmup that ramps the rate up, then a cosine decay that winds it down) and adaptive methods like Adam instead of one fixed number. The optimization page owns those methods; the point here is narrower and durable: the learning rate is one hyperparameter, and getting it wrong by a factor of two can be the difference between a model and a pile of NaNs.

The other knobs that decide a run

The learning rate is the most consequential hyperparameter, not the only one. A handful of others shape every serious training run:

• Batch size — how many examples the model looks at before each weight update. Larger batches give a smoother, more reliable gradient but need proportionally more memory and, past a point, generalize slightly worse; it also interacts with the learning rate, so the two are usually tuned together rather than separately.
• Epochs — how many times training passes over the whole dataset. Too few and the model underfits; too many and it starts memorizing the training set instead of the pattern.
• Regularization strength — the weight decay coefficient, or the dropout rate, that penalizes complexity to combat overfitting. This is a hyperparameter that directly trades training accuracy for generalization, and regularization is the page that unpacks it.
• Architecture — the number of layers and their width. These define how many parameters exist, which is why they are the least gradient-like of all: you cannot smoothly adjust them mid-run.

Every one of these is set before training and held fixed throughout, and every one interacts with the others — which is what makes choosing them a search problem rather than a lookup.

Types

Because you cannot differentiate your way to good hyperparameters, you search for them: pick some combinations, train a model with each, and keep whichever scores best on a held-out validation set (the reason a validation set is separate from the test set is exactly this — see cross-validation). There are three established search strategies, and they form a genuine progression from blind to informed.

Grid search picks a fixed set of values for each hyperparameter and tries every combination — a 3×3 grid over two hyperparameters is nine training runs. It is exhaustive and trivial to parallelize, and it is reliable in one or two dimensions. Its cost, though, is exponential: add a third hyperparameter with three values and you are at 27 runs, a fourth and you are at 81.

Random search draws each combination at random from the same ranges. It sounds worse and is usually better. Bergstra and Bengio, in "Random Search for Hyper-Parameter Optimization" (Journal of Machine Learning Research, vol. 13, pp. 281–305, 2012), showed both empirically and theoretically that "random search over the same domain is able to find models that are as good or better within a small fraction of the computation time." The reason is a picture worth internalizing. Suppose two hyperparameters, but only one of them actually matters (their Figure 1 makes it the horizontal axis). A 3×3 grid spends nine expensive training runs testing that important hyperparameter in only three distinct places, because the grid repeats each of its three values across the wasted axis. Nine random points test it in nine distinct places. Their words: "With grid search, nine trials only test g(x) in three distinct places. With random search, all nine trials explore distinct values of g." Since, as their Gaussian-process analysis found, "for most data sets only a few of the hyper-parameters really matter, but … different hyper-parameters are important on different data sets," grid search's evenly-spaced coverage is spent mostly on axes that do not count. This is not a close call: purely random search matched a careful manual-plus-grid tuning of deep belief networks on four of seven datasets and beat it on one, over a 32-dimensional space.

Bayesian optimization goes one step further and learns from the trials it has already run. It builds a cheap surrogate model of "hyperparameters → validation score," uses it to guess which untried combination is most promising, trains that one, updates the surrogate, and repeats. Where grid and random search are non-adaptive — every trial is chosen with no memory of the last — Bayesian methods spend later trials near the settings that looked good early. This is the engine inside most modern tuning tools, and it is what the Bergstra–Bengio paper explicitly frames random search as the baseline for: "a natural baseline against which to judge progress in the development of adaptive (sequential) hyper-parameter optimization algorithms." Related bandit-style methods — successive halving and Hyperband — attack the same problem from the cost side, starting many configurations cheaply and killing the losers early.

Real-World Applications

Every scikit-learn tutorial that tunes a model uses these by name. GridSearchCV and RandomizedSearchCV are among the library's most-used classes, and the choice between them is the Bergstra–Bengio result made into an API decision: for anything past two or three hyperparameters, the documentation and the field both point to random.

Kaggle competitions are frequently decided by hyperparameter tuning. When the top of a leaderboard is a wall of gradient-boosted trees (XGBoost, LightGBM), the models are near-identical and the separation comes from tuning learning rate, tree depth, and regularization — which is why tools like Optuna (Preferred Networks) and Ray Tune became standard competition equipment. Their default samplers are Bayesian, precisely because compute per trial is high and blind search wastes it.

Cloud platforms sell hyperparameter search as a managed service. Google's internal Bayesian optimization service, Vizier (Golovin et al., KDD 2017), tunes hyperparameters across Google and is exposed to customers as Vertex AI Vizier; Weights & Biases Sweeps and Amazon SageMaker's tuner occupy the same slot. The fact that a whole product category exists to run this search is the clearest evidence of its cost: each "trial" is a full model training run, and a sweep is dozens or hundreds of them.

At frontier scale the search is done on small models and extrapolated. You cannot run hundreds of trials on a model that costs millions of dollars to train once, so labs tune hyperparameters on small proxies and transfer the settings up — the practical partner of the scaling laws that predict how the rest of the run behaves. The same discipline applies far below the frontier: whenever you fine-tune an existing model, the learning rate and epoch count are hyperparameters you are choosing, and choosing badly there overfits a small dataset just as fast.

Key Concepts

• The clean line: chosen before, versus learned during. If a number is fixed at the start of training and untouched by gradient descent, it is a hyperparameter; if training moves it, it is a parameter. The learning rate is chosen; the weights are learned. This one test resolves almost every "which is it" question.
• A hyperparameter is a ceiling, not a nudge. A bad weight is fixed by the next gradient step. A bad learning rate is never fixed by training at all — it silently limits the best loss the run can reach, which is why tuning happens in an outer loop of repeated training runs.
• Search, don't solve. Because there is no gradient to follow, you find good hyperparameters by trying combinations and scoring them on validation data — and random search beats grid search for all but the lowest-dimensional problems.

Challenges

Each trial is a whole training run, so the search is the expensive part. Tuning is not a tweak you add at the end; it multiplies your compute bill by the number of configurations you try. A 100-trial sweep is 100 trainings. This is the single fact that shapes every practical decision in the field — it is why random beats grid (you get more information per expensive trial), why Bayesian methods exist (spend trials where they help), and why frontier labs tune on proxies instead.

Tuning on the test set quietly launders the answer. Every time you pick hyperparameters by their score on a held-out set, that set has leaked a little information into the model. Do it enough times against the same validation set and you overfit to it — your reported number looks great and does not survive contact with genuinely new data. This is why a separate, untouched test set exists, and why the honest reporting of a tuned model is harder than it looks. It is the bias–variance tradeoff wearing a different hat.

Good hyperparameters do not transfer. The Bergstra–Bengio finding cuts both ways: because different hyperparameters matter on different datasets, the settings someone published for their problem are a starting guess, not an answer, for yours. A learning rate that was perfect at one batch size or model width is often wrong at another, so copied configurations regularly disappoint — the search usually has to be at least partly redone.

They interact, so tuning them one at a time misleads. The best batch size depends on the learning rate; the best regularization strength depends on how many epochs you run. Optimizing each knob independently finds a combination that is jointly worse than what a joint search would find, which is the whole reason grid and random search vary everything at once.

Future Trends

The clearest direction is making hyperparameters transferable so the expensive search happens once, cheaply. Techniques for reparametrizing a network so that its best learning rate is stable as the model grows (the "μTransfer" line of work) let a lab tune a small model and reuse the settings on a large one — turning what was a per-model search into a per-family one. This matters most exactly where trials are most expensive, at the frontier.

The second is the steady automation of the search itself under the banner of AutoML: pipelines that treat the choice of model, features, and hyperparameters as one joint optimization. The Bergstra and Bengio paper anticipated this, naming random search as "a natural baseline against which to judge progress" for the adaptive methods that followed — and every serious AutoML system still reports against that baseline, because a method that cannot beat random draws is not earning its complexity. The honest through-line is that none of this removes the need to choose: it only moves the choosing from a human typing numbers to an algorithm searching for them, and the reason both exist is the same one this whole page turns on — there is no gradient pointing the way.