Term tag: neurons

Activation Functions

Without a non-linearity, stacked layers collapse into one matrix. How activation functions fix that, and why ReLU, GELU and SwiGLU are the defaults.

Layers in Neural Networks

A layer is one weight matrix plus a nonlinearity. How it differs from a neuron and a parameter, why stacking helps, and how many layers real models have.

Neural Network

Why a stack of layers does what one cannot: two linear layers collapse into a single matrix, and a hidden layer of two units solves XOR.

Neurons

One neuron multiplies its inputs by weights, adds a bias and applies a function. The arithmetic worked by hand, and where the brain analogy breaks.