Term tag: attention mechanism

FlashAttention

The exact-attention algorithm that made long context affordable: it never writes the n x n score matrix to GPU memory, so it runs faster and changes no output.

Grouped-Query Attention (GQA)

Grouped-query attention lets query heads share key/value heads, shrinking the KV cache that dominates long-context inference at near multi-head quality.

Positional Encoding (RoPE)

Positional encoding tells a transformer where each token sits in a sequence. Why self-attention needs it, and how rotary embeddings (RoPE) work.

Self-Attention

The transformer operation that lets every token in a sequence weigh every other token by relevance, computed as softmax(QKᵀ/√d_k)·V.

State-Space Models (Mamba)

State-space models are sequence models that scale linearly with length, not quadratically like attention. Mamba's input-dependent state made them competitive.

Transformer

The 2017 neural-network architecture that processes a whole sequence in parallel using self-attention instead of recurrence — the basis of modern LLMs.

Vision Transformer (ViT)

A Vision Transformer (ViT) cuts an image into fixed patches, embeds each as a token, and runs a transformer encoder — trading CNN inductive bias for scale.