Context Window

Definition

A language model has no memory. The context window is the only thing it can see — and it re-reads all of it, from scratch, every single time it answers.

Everything competes for that one budget: the system instructions, the tool definitions, every message in the conversation so far, any document you pasted, and the reply as it is being written. It is measured in tokens, and the exchange rate measured on this site's own glossary prose is 1.29 tokens per English word. So the million-token windows the frontier labs advertise in 2026 hold roughly 775,000 words — seven or eight novels, or a mid-sized codebase.

The number on the spec sheet is not the number you can use. On tasks that require finding and combining several facts spread through a long document, frontier models hold up best somewhere under 200,000–400,000 tokens, even when the advertised maximum is five times that. A large window does not mean the model reads all of it equally well.

How It Works

The model is stateless — every turn resends everything

This is the mechanism almost everything else follows from. The API has no session; turn 20 of a conversation is a fresh request containing turns 1 through 19 verbatim. The model rebuilds its entire understanding from the text you send, then throws it away.

Watch what that costs. Take a chat that grows by 500 tokens per turn. Turn 1 bills 500 input tokens, turn 2 bills 1,000, turn 20 bills 10,000. Across twenty turns:

500 x (1 + 2 + ... + 20) = 500 x 210 = 105,000 input tokens billed

for a conversation that is only 10,000 tokens long. You paid for it 10.5 times over. The conversation itself grows linearly; the bill grows with the square of the turn count, and the gap between those two curves is the overpayment. Nothing is broken — that is simply what statelessness costs, and it is why prompt caching exists as a product.

Why the window ends where it does

Two costs scale with length, and they are not the same cost. Attention compute grows with the square of the sequence: every token looks at every earlier token. The KV cache — the keys and values the model stores so it never re-reads the past — grows linearly. Efficient attention kernels such as FlashAttention took the quadratic memory term off the table, so what binds in practice is the linear one.

Run it on a real model. Meta's Llama 3.1 70B stores 320 KiB per token. At a 128K context that is 43 GB for one user. Scale it to a million tokens:

327,680 bytes x 1,000,000 tokens = 328 GB - for a single session

An eight-GPU H100 server has 640 GB of memory in total, and 141 GB of that is already spent holding the model's weights. The 499 GB left over has room for exactly one user at a million tokens. A million-token window is a memory problem long before it is an attention problem, which is why the engineering underneath one is overwhelmingly cache engineering — and why context length, price, and concurrency limits move together across every provider. The broader version of that constraint belongs to the memory wall.

The cheap escape is sliding-window attention: let each token attend only to the last few thousand tokens rather than all of them, which makes memory constant instead of linear. Mistral 7B and Google's Gemma models use it. What you trade away is exactly the thing a long window was for — genuine long-range recall. Llama 4 Scout takes the opposite route to reach 10 million tokens, interleaving attention layers that carry no positional embeddings at all with rotary-embedded ones, plus inference-time attention scaling.

What happens when you run out

Three different behaviours, and confusing them causes real bugs:

• A direct API call errors. Exceed the window and the request is rejected. Nothing is silently dropped.
• A chat interface drops or summarizes history. The oldest turns fall out of view or get compressed to make room. No error, no warning — this is the "the AI forgot" experience, and it is a context-window limit, not a memory feature.
• An agent framework compacts. Long-running agents summarize their own history mid-run and continue. Claude Opus 5 ships compaction and context editing as API features precisely because agents outlive their windows.

Real-World Applications

Coding agents that hold a repository. An agent working across a codebase keeps file contents, tool schemas, and its own trail of edits in the window at once. It is the workload that exhausts a window fastest, and the one where compaction stops being optional — a two-hour session generates far more text than any window holds, however large.

Prompt caching, and what it is worth. Because the whole prefix is re-sent every turn, providers let you keep it warm: Anthropic bills cached prefix tokens at 0.1x the normal input rate and charges 1.25x to write the cache. Put an agent with a 50,000-token codebase prefix through 100 turns in an hour, at $5 per million input tokens:

• Without caching: 100 x 50,000 = 5,000,000 input tokens → $25 per hour
• With caching: one write at 1.25x ($0.31) plus 99 reads at 0.1x ($2.48) → $2.79 per hour

Roughly nine times cheaper, for no change other than putting the stable content first and the variable content last — a prompt layout rule that falls straight out of the mechanism above.

Document review. Whole contracts, filings and research papers now fit in one request, removing the chunking machinery an earlier generation of tools needed. What it does not remove is the accuracy problem below — summarizing a document means combining facts scattered through it, which is precisely the workload long context is weakest at.

Choosing between a big window and retrieval. Pasting everything is simpler, costs more per request, and degrades as the pile grows. Retrieval-augmented generation pulls in only the relevant passages and costs less, at the price of a retrieval stack you maintain. For one document that fits comfortably, use the window; for a corpus, retrieve — RAG vs long context covers how to read the benchmarks behind that call.

Key Concepts

Input budget and output budget are different numbers. The advertised window is almost entirely input, and the cap on what the model can write back is an order of magnitude smaller:

These are advertised maximums, not effective ones. On reasoning models the internal thinking is spent from the output budget too, so a long chain of reasoning leaves less room for the visible answer.

Context is not memory. The window is per-conversation and vanishes when the chat ends. "Memory" in an assistant is a separate store that persists across sessions and injects a few relevant facts back into the window when needed. Different mechanisms, different expectations; context window vs tokens vs memory untangles all three.

Windows can shrink. Grok 4.5 halved its predecessor's window, from 1,000,000 tokens down to 500,000, while raising the price — and Grok 4.3 remains available specifically as the long-context option. Context length is a product decision with a cost behind it, not a capability ratchet that only moves up.

Long context can carry a price premium. Google's Gemini 3.1 Pro charges $2.00 in / $12.00 out per million tokens for prompts up to 200K, and $4.00 / $18.00 above that — the same tokens, priced differently by how many of them there are. Claude Opus 5 charges standard rates across its full 1M window. Check which model you are on before budgeting long prompts.

Challenges

Accuracy decays inside the window — "context rot". As a window fills, the model's ability to reliably use what is in it drops, and the standard benchmark hides this. Single-needle "find the planted sentence" tests score above 90% at a million tokens for nearly every frontier model. Multi-needle tests — find eight facts in different places and combine them — fall off a cliff. Pick a model on the multi-needle numbers; the headline number tells you almost nothing about the work you will actually give it.

Teams budget for the context and get billed for the turns. A twenty-turn chat bills 10.5x its own length, so the surprise arrives as an invoice, not as an error. Caching is the fix, and it only works when the stable part of the prompt is byte-identical every time — a timestamp in the system prompt silently invalidates the entire prefix.

Latency scales with prompt length, not answer length. Before the first character appears the model must read the entire prompt — the prefill phase — and that work is proportional to how much you sent. A million-token request spends seconds reading before it emits anything, on every turn. A user waiting on a long-context request is waiting on tokens they wrote, not tokens the model is writing.

More context can make the answer worse. Irrelevant text in the window is not neutral: the model cannot tell your carefully chosen document from the four you added "just in case", and a bigger window is a standing invitation to stuff it. A smaller, curated context routinely beats a large noisy one — which is why prompt engineering for long contexts is mostly about deciding what to leave out.

Vendor numbers are not comparable. 10M via interleaved positional encoding, 1M via a dense window, and 500K priced as a premium tier are three products wearing the same unit. Nothing forces a lab to publish the length at which quality holds, and none of them volunteer it.

Future Trends

The competition is shifting from advertised size to effective length. Once every frontier model claims about a million tokens, the number differentiates nothing, and the benchmarks that matter become the ones measuring whether the model can use the window — multi-fact retrieval and long-document summarization rather than single-needle recall.

Context management is becoming an API primitive. Compaction and context editing now ship as first-class API features. The direction of travel is that "what stays in the window" becomes a configurable policy the provider executes, not a summarization loop each team writes badly on its own.

The cache is turning into a storage tier. Prefix caches shared across users and requests, spilled to disk and pulled back when a conversation resumes, are already how large agent deployments avoid re-reading the same codebase a hundred times an hour. Once the cache is a durable, addressable object, the boundary between "context" and "memory" starts to dissolve — and the model's working set stops being something it rebuilds from nothing every time you say hello.

Code Example

Measure before you send. Token counts are model-specific, so use the model's own counter rather than a generic tokenizer — and ask the API for the window instead of hardcoding a number that will change.

from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-opus-5"

# The window is a published property of the model — look it up, don't hardcode it.
limit = client.models.retrieve(MODEL).max_input_tokens

conversation = [
    {"role": "user", "content": open("contract.txt").read()},
]

used = client.messages.count_tokens(model=MODEL, messages=conversation).input_tokens

print(f"{used:,} / {limit:,} tokens — {used / limit:.1%} of the window")

if used > limit * 0.4:
    print("Past the comfortable range. Retrieve the relevant passages instead of pasting everything.")

The 40% threshold is a judgement call, not a law — it reflects effective context sitting well below the advertised maximum. Move it based on how much your task depends on combining facts scattered through the text.