Gradients and Derivatives: Backpropagation Deep Dive

To fully understand backpropagation, we need to explore how gradients flow using calculus.

---

🧮 Derivatives and Chain Rule

The chain rule is key to computing gradients in neural networks:

∂L/∂x = ∂L/∂z × ∂z/∂x

Where L is the loss, z is an intermediate variable, and x is a weight or activation.

In neural networks, we apply this recursively through layers.

---

🔁 Example: Two-layer Network

Let’s say we have:

z₁ = w₁x + b₁ → a₁ = ReLU(z₁)
z₂ = w₂a₁ + b₂ → ŷ = sigmoid(z₂)

Loss: L = MSE(ŷ, y)

To compute the gradient of L w.r.t. w₁:

1. ∂L/∂ŷ
2. ∂ŷ/∂z₂
3. ∂z₂/∂a₁
4. ∂a₁/∂z₁
5. ∂z₁/∂w₁

---

📉 Matrix Form

In vectorized networks, backpropagation uses matrix calculus.

∂L/∂W = ∂L/∂Z × ∂Z/∂W

Frameworks like PyTorch or TensorFlow handle this automatically using autograd.

---

🧠 Intuition

Backprop tells us how a small change in a weight affects the final loss.

The goal of gradient descent is to follow the negative gradient to minimize loss.

---

⚙️ Gradient Flow Issues

• Vanishing gradients: Sigmoid/Tanh cause very small derivatives
• Exploding gradients: Can happen in deep networks
• Solutions: ReLU, LayerNorm, Residual connections

---

📊 Visualization

This shows how errors move backward and accumulate.

---

Self-Check

• What does the chain rule allow us to compute?
• What causes vanishing gradients?
• How do frameworks calculate gradients?