Term tag: activation functions

Activation Functions

Without a non-linearity, stacked layers collapse into one matrix. How activation functions fix that, and why ReLU, GELU and SwiGLU are the defaults.

Logistic Regression (LR)

Logistic regression turns weighted evidence into a probability with the sigmoid: why the name says regression, how to read coefficients, and when to use it.

Neural Network

Why a stack of layers does what one cannot: two linear layers collapse into a single matrix, and a hidden layer of two units solves XOR.

Neurons

One neuron multiplies its inputs by weights, adds a bias and applies a function. The arithmetic worked by hand, and where the brain analogy breaks.

Normalization (LayerNorm, RMSNorm, BatchNorm)

Normalization rescales a layer's activations to a stable mean and scale so deep networks train faster. How BatchNorm, LayerNorm and RMSNorm differ.