Definition
In machine learning, optimization means minimizing a loss function: adjusting a model's parameters so that a single number — the loss, which measures how wrong the model's predictions are — comes out as small as possible. That is not one task among many that a model does; it is the whole of training. A freshly initialized network is just a pile of random numbers, and "learning" is the process of an optimization algorithm nudging those numbers, over and over, until the loss stops falling.
The reason this needs an algorithm at all, rather than a formula, is that you almost never get to solve for the answer directly. For a model with a billion parameters the loss is a surface in a billion dimensions with no closed-form bottom, so the optimizer cannot jump to the minimum — it can only feel the slope where it currently stands and take a step downhill. It repeats that step millions of times. The value you ship is a good point on that surface, reached by iteration, not the provably-best point, which for a network of any size is out of reach.
This is training-time optimization, and it is a different subject from inference optimization, which is about making an already-trained model run faster and cheaper. This page is about the fitting: how a loss gets minimized and what that costs.
How It Works
Every optimization has three pieces: an objective (the loss you want small), the parameters you are allowed to change, and a step rule for changing them. The engine that connects them is the gradient — the vector of partial derivatives of the loss with respect to each parameter. Each entry answers one question: if I increase this one parameter a little, does the loss go up or down, and how steeply? Move every parameter a small amount in the opposite direction of its own gradient entry, and the loss goes down. That single move, repeated, is gradient descent, and the gradient itself is computed by backpropagation.
It helps to watch one step on the smallest possible model: a single weight w that should turn an input x = 2 into an output y = 6, so the right answer is w = 3. The loss is the squared error, L(w) = (w·x − y)². Start at a bad guess, w = 1. The prediction is 2, the error is −4, and the loss is 16. The gradient is dL/dw = 2·(w·x − y)·x = 2·(−4)·2 = −16 — negative, meaning the loss falls if w rises. With a step size (learning rate) of 0.05, the update is
w ← w − 0.05 · (−16) = 1 + 0.8 = 1.8
and the loss drops from 16 to 5.76, a 64% cut in a single step. Nothing about that move consulted where the minimum is; it only used the slope underfoot, which is why one step is never enough and the process is inherently iterative.
The learning rate is the one hyperparameter that decides whether iteration works at all. Too small and the loss crawls; too large and each step overshoots the bottom and the loss climbs instead of falls. For the example above there is an exact threshold — steps larger than 0.25 diverge — and the gradient descent page works the full set of regimes out on a table. The point for optimization is that the same fixed step size is rarely right for the whole run, which is why real training uses a learning-rate schedule: a short warm-up, then a slow decay, so the optimizer takes bold steps early and careful ones near the bottom.
Convex versus non-convex
The example above has a convex loss — one bowl, one bottom, and any downhill path reaches it. If every loss were convex, optimization would be nearly solved. Neural-network losses are non-convex: a landscape of many valleys at different depths. A global minimum is the deepest point anywhere; a local minimum is the bottom of a valley that is lowest only in its own neighborhood. Gradient descent, being blind beyond its current position, settles into whichever valley it happened to be standing over. In practice this matters less than it sounds: for a loss over a billion parameters, a point that curves upward in every one of a billion directions is astronomically rarer than a saddle point that curves up in some directions and down in others, so training rarely gets trapped in a bad local minimum — it contends mostly with saddles and long flat plateaus where the gradient is nearly zero and progress stalls.
The optimizers people actually use
Plain gradient descent using every training example to compute one gradient is too slow, so real training uses stochastic gradient descent (SGD): estimate the gradient from a small random batch and step on that noisy estimate. The noise is a feature — it lets the optimizer rattle out of shallow traps. Three refinements dominate modern practice, and they differ in how much extra state they carry per parameter:
- SGD with momentum keeps one running average of past gradients — a velocity — so the optimizer builds speed along consistent directions and damps the zig-zag across narrow valleys. That is one extra number per parameter.
- Adam (Kingma & Ba, 2014) keeps two running averages per parameter: a first moment
m(the mean of recent gradients) and a second momentv(the mean of recent squared gradients). It divides each step by the square root ofv, giving every parameter its own effective learning rate — large where gradients have been small and noisy, small where they have been large. The paper's recommended defaults, still the near-universal starting point, are a step size of0.001withβ₁ = 0.9,β₂ = 0.999, andε = 10⁻⁸. - AdamW (Loshchilov & Hutter, 2019) is Adam with weight decay applied separately from the gradient step rather than folded into it, and it is the default optimizer for training essentially every large transformer.
That second moment is not free. Because Adam stores m and v, its optimizer state is two extra full-size copies of the model, on top of the parameters themselves and their gradients. Count the copies for a model with N parameters trained in 32-bit floats: parameters (4N bytes) + gradients (4N) + m (4N) + v (4N) = 16 bytes per parameter, four copies of the parameter array resident at once. A one-billion-parameter model therefore needs on the order of 16 GB just for these buffers to train. Running the same model for inference holds a single copy of the weights, often in 16-bit or 8-bit precision. This is the concrete reason training a model demands far more memory than deploying it: the answer you want is one array of weights, but producing it requires the optimizer to carry three more arrays beside it the entire time.
Real-World Applications
The dominant application is pretraining large language models. Models in the GPT, Llama, and PaLM families are trained by AdamW minimizing a next-token cross-entropy loss over trillions of tokens, with a warm-up-then-cosine-decay learning-rate schedule and gradient clipping to keep a single bad batch from blowing up the run. The optimizer choice is not incidental: at that scale the m and v buffers are large enough that engineers shard them across GPUs (the standard trick in ZeRO and FSDP training) precisely because Adam's optimizer state is the memory that does not fit.
In computer vision, convolutional networks such as ResNet were for years trained with SGD with momentum rather than Adam, because on those architectures the simpler optimizer generalized slightly better for the same accuracy — a reminder that the "best" optimizer is empirical, not settled. Beyond deep learning, the same minimize-an-objective machinery is what fits the weights in logistic regression, positions the boundary in a support-vector machine, and drives classical training loops throughout machine learning.
Challenges
The objective is a proxy, and the optimizer takes it literally. The thing you can differentiate is rarely the thing you want. Accuracy, revenue, and user satisfaction are not smooth functions of the weights, so you minimize a stand-in — cross-entropy, or watch time — and the optimizer delivers exactly what you asked for rather than what you meant. A recommender minimizing "negative watch time" will discover that outrage holds attention. Nothing failed in the optimizer; the objective was wrong, and no amount of better optimization fixes a mis-specified loss.
Saddle points and plateaus, not local minima, are the real obstacle. Because bad local minima are so rare in very high dimensions, the practical enemy is the flat region where the gradient is tiny in every direction and steps shrink to nothing. This is what momentum and Adam's per-parameter scaling are built to escape, and it is why a run can appear stuck for thousands of steps and then resume falling.
The learning rate has no safe default. Too high and the loss diverges to NaN, usually mistaken for a code bug when it is a step-size bug; too low and a run that should finish in an afternoon takes a week to reach the same place. The stable range depends on the loss surface, which changes as training proceeds, which is why schedules and adaptive methods exist rather than a single magic number.
Optimizer state is a first-order cost, not a footnote. The memory to train is dominated by the optimizer's per-parameter buffers, so switching from Adam to a memory-lighter optimizer, or sharding the state across devices, is often what decides whether a model fits on the hardware at all — a constraint that shapes which models get built.
Code Example
A complete optimizer in eight lines: minimize the squared-error loss for the one-weight model from above, w·x = y with x = 2, y = 6, so the target is w = 3. Each pass computes the loss, its gradient, and takes one gradient-descent step.
x, y = 2.0, 6.0 # target weight is y/x = 3.0
w = 1.0 # starting guess
lr = 0.05 # learning rate
for step in range(5):
pred = w * x
loss = (pred - y) ** 2
grad = 2 * (pred - y) * x # dL/dw
print(f"step {step}: w={w:.4f} loss={loss:.4f} grad={grad:.4f}")
w = w - lr * grad # the optimization step
print(f"final: w={w:.4f} (target 3.0)")
Running it prints the loss falling by a constant factor every step as w climbs toward 3:
step 0: w=1.0000 loss=16.0000 grad=-16.0000
step 1: w=1.8000 loss=5.7600 grad=-9.6000
step 2: w=2.2800 loss=2.0736 grad=-5.7600
step 3: w=2.5680 loss=0.7465 grad=-3.4560
step 4: w=2.7408 loss=0.2687 grad=-2.0736
final: w=2.8445 (target 3.0)
A real network replaces w with hundreds of billions of parameters and the hand-written grad with backpropagation, but the loop is the same three lines: measure the loss, get the gradient, step. Swap the last line for Adam and you would carry two extra arrays, m and v, updated alongside w — the same buffers that make training memory-hungry.