Term tag: Deep Learning

Activation Functions

Without a non-linearity, stacked layers collapse into one matrix. How activation functions fix that, and why ReLU, GELU and SwiGLU are the defaults.

AI Research

AI research is the systematic investigation into the development of algorithms, models, and systems that exhibit intelligent behavior.

Artificial Intelligence (AI)

AI is software whose behaviour is fitted from data instead of written rule by rule: what that means, why the label keeps moving, and where you already use it.

Attention Mechanism

Each token emits a query, key and value; dot-product scores scaled by √d become softmax weights over every other token. The cost grows as n².

Autoencoder

A neural network trained to reproduce its own input through a bottleneck too narrow to carry it — so the compressed code, not the copy, is the product.

Backpropagation

Backpropagation computes the gradient of the loss for every parameter at once, for about the cost of one extra forward pass. How the chain rule does it.

Calibration

Calibration means a model's confidence matches reality: of everything it calls 80% likely, about 80% should happen — a property separate from raw accuracy.

Classifier-Free Guidance (CFG)

Classifier-free guidance (CFG) is how a diffusion model follows a prompt: each step runs the model with and without the prompt and extrapolates the gap.

Computer Vision (CV)

How computers turn a grid of pixel values into labels, boxes and masks — the task ladder, the 2012 hinge, and where vision is reliable versus brittle.

Convolution

Convolution slides a small kernel over an image, multiplying and summing each patch into one output number. The arithmetic of stride, padding and kernels.

Convolutional Neural Network (CNN)

The architecture that assumes image features are local and position-independent — how the stack builds a hierarchy, and where CNNs still beat transformers.

Deep Learning

What makes deep learning deep: each layer builds on the one below, so capacity grows multiplicatively with depth and only additively with width.

Diffusion Language Models (DLMs)

Text models that unmask many tokens per pass instead of writing one at a time — what that actually buys in speed, and what it costs in quality.

Diffusion Model

A generative model that learns to undo a fixed noising process — what the network really predicts, why the step count is the cost, and why it displaced GANs.

Federated Learning

Federated learning trains a shared model across many devices without moving their raw data: the local-train, send-updates, aggregate loop.

Generalization

Why a model that scores 99% in training fails in the world: the generalization gap, the test-set size needed to measure it, and why classical theory is wrong.

Generative Adversarial Network (GAN)

Two networks trained against each other: one invents data, the other judges it. The generator never sees a real example — only the judge's verdict.

Generative AI

Generative AI learns the probability distribution its training data came from, well enough to draw new samples from it — not just sort inputs into classes.

GPU Computing

Why a graphics chip runs AI: a GPU spends its transistors on arithmetic instead of on making one thread fast — and what that trade wins and costs.

Gradient Descent

The rule that turns a gradient into a weight update. Worked on f(x)=x²: why the step size alone decides between converging, oscillating and blowing up.

Graph Neural Networks (GNN)

A GNN takes adjacency as data, not as an architectural assumption. Message passing worked by hand, why k layers reach d^k nodes, and why deep GNNs collapse.

Knowledge Distillation

How a small model learns what a big one knows: the teacher's full probability distribution carries structure the hard labels do not — Hinton's dark knowledge.

Layers in Neural Networks

A layer is one weight matrix plus a nonlinearity. How it differs from a neuron and a parameter, why stacking helps, and how many layers real models have.

Matrix Multiplication (GEMM)

The one operation that consumes most of an AI model's compute — and the reason AI chips are built the way they are, from tensor cores to systolic arrays.

Model Compression

Making a trained model smaller and cheaper to run — quantization, pruning, distillation and low-rank methods, and which one your problem actually needs.

Model Size

Model size is a model's parameter count. What that one number predicts — memory, cost per token, capability — worked on real models from 110M to trillions.

Multimodal AI

How one model reads an image and text at once: the picture is cut into fixed patches, each patch becomes a token, and those tokens cost context.

Neural Network

Why a stack of layers does what one cannot: two linear layers collapse into a single matrix, and a hidden layer of two units solves XOR.

Neurons

One neuron multiplies its inputs by weights, adds a bias and applies a function. The arithmetic worked by hand, and where the brain analogy breaks.

Normalization (LayerNorm, RMSNorm, BatchNorm)

Normalization rescales a layer's activations to a stable mean and scale so deep networks train faster. How BatchNorm, LayerNorm and RMSNorm differ.

One-shot Learning

Learning a new class from exactly one example — and the split that matters: one-shot prompting updates no weights; classical one-shot training does.

Optimization

In machine learning, optimization means minimizing a loss function to fit a model's parameters — the iterative, gradient-driven mechanism behind training.

Parallel Processing

Running many computations at the same instant across many processing units. In AI it makes matrix multiplication fast, which is why GPUs train models.

Parameters

The learned numbers inside a model — what '70 billion parameters' actually means, and how the count converts directly into the memory a GPU must hold.

Pattern Recognition (PR)

Pattern recognition is the automatic discovery and classification of regularities in data — the older name for what became statistical machine learning.

Pooling

A downsampling operation in CNNs that shrinks a feature map by summarizing each small region into one value, cutting resolution and compute.

Positional Encoding (RoPE)

Positional encoding tells a transformer where each token sits in a sequence. Why self-attention needs it, and how rotary embeddings (RoPE) work.

Protein Folding

How a chain of amino acids collapses into the 3D shape that sets a protein's function - and how predicting that shape from sequence became AlphaFold's win.

Quantization

Running a model in fewer bits — FP16, FP8, INT4 — to cut memory and cost, what accuracy it actually costs, and why a memory-bound model also gets faster.

Recurrent Neural Network (RNN)

A recurrent neural network reads a sequence one element at a time, carrying a hidden state that summarizes everything seen so far to model order and context.

Representation Learning

Representation learning is machine learning that discovers the useful features of raw data automatically, instead of humans hand-engineering them.

Residual Connections (Skip Connections)

Residual connections add a block's input back to its output — F(x) + x — the trick that lets gradients flow through deep nets and made transformers trainable.

Robustness

A model's ability to keep performing when inputs are noisy, out of distribution, or crafted by an attacker — not just accurate on clean test data.

Scaling Laws

The empirical curves that make AI capex rational: model loss falls predictably with compute, data and parameters — and what they do not promise.

Self-Attention

The transformer operation that lets every token in a sequence weigh every other token by relevance, computed as softmax(QKᵀ/√d_k)·V.

Speech-to-Speech

Speech-to-speech AI takes audio in and returns audio out — conversation or translation — via a cascade (ASR to text to TTS) or one end-to-end model.

State-Space Models (Mamba)

State-space models are sequence models that scale linearly with length, not quadratically like attention. Mamba's input-dependent state made them competitive.

Tensor Operations

The array manipulations — elementwise math, matrix multiplication, reshaping, broadcasting, reduction — that every neural network is built out of.

Test-Time Compute

Spending compute at inference instead of training — how reasoning models buy accuracy with thinking time, and why it moved chip demand into serving.

Training

Training fits a model's parameters to data: forward pass, measure the loss, backward pass, nudge every weight downhill, repeat until the loss stops falling.

Transformer

The 2017 neural-network architecture that processes a whole sequence in parallel using self-attention instead of recurrence — the basis of modern LLMs.

Vision Transformer (ViT)

A Vision Transformer (ViT) cuts an image into fixed patches, embeds each as a token, and runs a transformer encoder — trading CNN inductive bias for scale.

Voice Recognition

How AI turns spoken words into text (speech recognition / ASR) — and how that differs from identifying who is speaking (speaker recognition).

Weights

A weight is a single learned number setting the strength of a connection between two neurons; together, the weights are the trained model.