Term tag: transformer

Attention Mechanism

Each token emits a query, key and value; dot-product scores scaled by √d become softmax weights over every other token. The cost grows as n².

FlashAttention

The exact-attention algorithm that made long context affordable: it never writes the n x n score matrix to GPU memory, so it runs faster and changes no output.

Grouped-Query Attention (GQA)

Grouped-query attention lets query heads share key/value heads, shrinking the KV cache that dominates long-context inference at near multi-head quality.

KV Cache

The memory an LLM keeps so it never recomputes the past — why a long context costs money, and how many users one GPU can actually serve at once.

Large Language Model (LLM)

A large language model predicts the next token of text. How that produces answers, where its knowledge is stored, and why an LLM is not a chatbot.

Layers in Neural Networks

A layer is one weight matrix plus a nonlinearity. How it differs from a neuron and a parameter, why stacking helps, and how many layers real models have.

Matrix Multiplication (GEMM)

The one operation that consumes most of an AI model's compute — and the reason AI chips are built the way they are, from tensor cores to systolic arrays.

Normalization (LayerNorm, RMSNorm, BatchNorm)

Normalization rescales a layer's activations to a stable mean and scale so deep networks train faster. How BatchNorm, LayerNorm and RMSNorm differ.

Positional Encoding (RoPE)

Positional encoding tells a transformer where each token sits in a sequence. Why self-attention needs it, and how rotary embeddings (RoPE) work.

Recurrent Neural Network (RNN)

A recurrent neural network reads a sequence one element at a time, carrying a hidden state that summarizes everything seen so far to model order and context.

Reranking (Cross-Encoder)

Reranking rescores the top-N results from a fast retriever with a slow, accurate cross-encoder that reads the query and document together.

Residual Connections (Skip Connections)

Residual connections add a block's input back to its output — F(x) + x — the trick that lets gradients flow through deep nets and made transformers trainable.

Self-Attention

The transformer operation that lets every token in a sequence weigh every other token by relevance, computed as softmax(QKᵀ/√d_k)·V.

Simple Continual Pretraining (SCP)

Method for turning a pretrained autoregressive LLM into a diffusion language model by continuing pretraining with a bidirectional attention mask.

State-Space Models (Mamba)

State-space models are sequence models that scale linearly with length, not quadratically like attention. Mamba's input-dependent state made them competitive.

Transformer

The 2017 neural-network architecture that processes a whole sequence in parallel using self-attention instead of recurrence — the basis of modern LLMs.

Vision Transformer (ViT)

A Vision Transformer (ViT) cuts an image into fixed patches, embeds each as a token, and runs a transformer encoder — trading CNN inductive bias for scale.