Term tag: Neural Networks

Activation Functions

Without a non-linearity, stacked layers collapse into one matrix. How activation functions fix that, and why ReLU, GELU and SwiGLU are the defaults.

AI Research

AI research is the systematic investigation into the development of algorithms, models, and systems that exhibit intelligent behavior.

Attention Mechanism

Each token emits a query, key and value; dot-product scores scaled by √d become softmax weights over every other token. The cost grows as n².

Autoencoder

A neural network trained to reproduce its own input through a bottleneck too narrow to carry it — so the compressed code, not the copy, is the product.

Backpropagation

Backpropagation computes the gradient of the loss for every parameter at once, for about the cost of one extra forward pass. How the chain rule does it.

Catastrophic Forgetting

Why fine-tuning makes a model worse at everything it used to do well — the mechanism inside shared weights, and what each fix actually costs.

Computer Vision (CV)

How computers turn a grid of pixel values into labels, boxes and masks — the task ladder, the 2012 hinge, and where vision is reliable versus brittle.

Deep Learning

What makes deep learning deep: each layer builds on the one below, so capacity grows multiplicatively with depth and only additively with width.

Generative Adversarial Network (GAN)

Two networks trained against each other: one invents data, the other judges it. The generator never sees a real example — only the judge's verdict.

Graph Neural Networks (GNN)

A GNN takes adjacency as data, not as an architectural assumption. Message passing worked by hand, why k layers reach d^k nodes, and why deep GNNs collapse.

Knowledge Distillation

How a small model learns what a big one knows: the teacher's full probability distribution carries structure the hard labels do not — Hinton's dark knowledge.

Layers in Neural Networks

A layer is one weight matrix plus a nonlinearity. How it differs from a neuron and a parameter, why stacking helps, and how many layers real models have.

Mixture-of-Experts (MoE)

The architecture behind "671B total, 37B active": hundreds of expert sub-networks, a router that picks a few per token, and a memory bill that never shrinks.

Neural Network

Why a stack of layers does what one cannot: two linear layers collapse into a single matrix, and a hidden layer of two units solves XOR.

Neurons

One neuron multiplies its inputs by weights, adds a bias and applies a function. The arithmetic worked by hand, and where the brain analogy breaks.

Normalization (LayerNorm, RMSNorm, BatchNorm)

Normalization rescales a layer's activations to a stable mean and scale so deep networks train faster. How BatchNorm, LayerNorm and RMSNorm differ.

Parameters

The learned numbers inside a model — what '70 billion parameters' actually means, and how the count converts directly into the memory a GPU must hold.

Pooling

A downsampling operation in CNNs that shrinks a feature map by summarizing each small region into one value, cutting resolution and compute.

Recurrent Neural Network (RNN)

A recurrent neural network reads a sequence one element at a time, carrying a hidden state that summarizes everything seen so far to model order and context.

Residual Connections (Skip Connections)

Residual connections add a block's input back to its output — F(x) + x — the trick that lets gradients flow through deep nets and made transformers trainable.

State-Space Models (Mamba)

State-space models are sequence models that scale linearly with length, not quadratically like attention. Mamba's input-dependent state made them competitive.

Tensor Operations

The array manipulations — elementwise math, matrix multiplication, reshaping, broadcasting, reduction — that every neural network is built out of.

Transformer

The 2017 neural-network architecture that processes a whole sequence in parallel using self-attention instead of recurrence — the basis of modern LLMs.

Vision Transformer (ViT)

A Vision Transformer (ViT) cuts an image into fixed patches, embeds each as a token, and runs a transformer encoder — trading CNN inductive bias for scale.

Weights

A weight is a single learned number setting the strength of a connection between two neurons; together, the weights are the trained model.