How Models Learn via Gradient Descent

Summary

Learn how gradient descent helps models improve step by step by minimizing the loss function.

basic
core-ai

Gradient descent is the core optimization algorithm used to train most machine learning models.

It helps the model learn by minimizing the loss function step by step.

🧗 Intuition

Imagine a hiker trying to descend a mountain (loss function) in the fog:

  • The height = error (loss)
  • The direction = gradient (slope)
  • Each step = model update

The hiker wants to reach the bottom — minimum error.

🧮 How It Works

At each step:

  1. Calculate the loss.
  2. Compute the gradient (slope of the loss curve).
  3. Take a step in the opposite direction of the gradient.
  4. Update the model's parameters.

🔧 Formula (Simplified)

Let θ be the model’s parameter.
Update rule:

θ = θ - η * ∇L(θ)

Where:

  • η is the learning rate
  • ∇L(θ) is the gradient of the loss function

📉 Example

  • Prediction too high? Decrease the weight.
  • Prediction too low? Increase the weight.

Over time, the model “nudges” itself to better performance.

⚠️ Learning Rate Matters

  • Too small → Slow learning
  • Too big → Might overshoot the minimum
  • Choose carefully!

🧠 Summary

| Concept | Meaning | |----------------|----------------------------------| | Gradient | Slope of the loss curve | | Descent | Move in direction of lower error | | Learning Rate | Size of the step | | Goal | Minimize the loss |

✅ Self-Check

  • What does gradient descent try to minimize?
  • Why is the learning rate important?
  • How do gradients help the model learn?