---
source: 'https://howaiworks.ai/glossary/context-window'
section: glossary
title: Context Window
description: >-
  Everything a model can see at once — why the advertised million tokens is not
  the usable number, and why you pay for the whole window on every turn.
tags:
  - Context Window
  - LLM
  - Long Context
  - Tokens
  - Inference
  - Prompt Engineering
category: Natural Language Processing
datePublished: '2025-09-29'
lastUpdated: '2026-09-06'
---

# Context Window

> Everything a model can see at once — why the advertised million tokens is not the usable number, and why you pay for the whole window on every turn.

## Definition

**A language model has no memory. The context window is the only thing it can see — and it re-reads all of it, from scratch, every single time it answers.**

Everything competes for that one budget: the system instructions, the tool definitions, every message in the conversation so far, any document you pasted, and the reply as it is being written. It is measured in [tokens](https://howaiworks.ai/glossary/token), and the exchange rate measured on this site's own glossary prose is 1.29 tokens per English word. So the million-token windows the frontier labs advertise in 2026 hold roughly **775,000 words** — seven or eight novels, or a mid-sized codebase.

The number on the spec sheet is not the number you can use. On tasks that require finding and combining several facts spread through a long document, frontier models hold up best somewhere under **200,000–400,000 tokens**, even when the advertised maximum is five times that. A large window does not mean the model reads all of it equally well.

## How It Works

### The model is stateless — every turn resends everything

This is the mechanism almost everything else follows from. The API has no session; turn 20 of a conversation is a fresh request containing turns 1 through 19 verbatim. The model rebuilds its entire understanding from the text you send, then throws it away.

Watch what that costs. Take a chat that grows by 500 tokens per turn. Turn 1 bills 500 input tokens, turn 2 bills 1,000, turn 20 bills 10,000. Across twenty turns:

```
500 x (1 + 2 + ... + 20) = 500 x 210 = 105,000 input tokens billed
```

for a conversation that is only **10,000 tokens long**. You paid for it **10.5 times over**. The conversation itself grows linearly; the bill grows with the *square* of the turn count, and the gap between those two curves is the overpayment. Nothing is broken — that is simply what statelessness costs, and it is why prompt caching exists as a product.

### Why the window ends where it does

Two costs scale with length, and they are not the same cost. [Attention](https://howaiworks.ai/glossary/attention-mechanism) compute grows with the *square* of the sequence: every token looks at every earlier token. The [KV cache](https://howaiworks.ai/glossary/kv-cache) — the keys and values the model stores so it never re-reads the past — grows *linearly*. Efficient attention kernels such as FlashAttention took the quadratic memory term off the table, so what binds in practice is the linear one.

Run it on a real model. Meta's Llama 3.1 70B stores **320 KiB per token**. At a 128K context that is 43 GB for one user. Scale it to a million tokens:

```
327,680 bytes x 1,000,000 tokens = 328 GB - for a single session
```

An eight-GPU H100 server has 640 GB of memory in total, and 141 GB of that is already spent holding the model's weights. The 499 GB left over has room for **exactly one** user at a million tokens. **A million-token window is a memory problem long before it is an attention problem**, which is why the engineering underneath one is overwhelmingly cache engineering — and why context length, price, and concurrency limits move together across every provider. The broader version of that constraint belongs to the [memory wall](https://howaiworks.ai/glossary/memory-wall).

The cheap escape is **sliding-window attention**: let each token attend only to the last few thousand tokens rather than all of them, which makes memory constant instead of linear. Mistral 7B and Google's Gemma models use it. What you trade away is exactly the thing a long window was for — genuine long-range recall. Llama 4 Scout takes the opposite route to reach **10 million tokens**, interleaving attention layers that carry no positional embeddings at all with rotary-embedded ones, plus inference-time attention scaling.

### What happens when you run out

Three different behaviours, and confusing them causes real bugs:

- **A direct API call errors.** Exceed the window and the request is rejected. Nothing is silently dropped.
- **A chat interface drops or summarizes history.** The oldest turns fall out of view or get compressed to make room. No error, no warning — this is the "the AI forgot" experience, and it is a context-window limit, not a memory feature.
- **An agent framework compacts.** Long-running agents summarize their own history mid-run and continue. [Claude Opus 5](https://howaiworks.ai/models/claude-opus) ships compaction and context editing as API features precisely because agents outlive their windows.

## Real-World Applications

**Coding agents that hold a repository.** An agent working across a codebase keeps file contents, tool schemas, and its own trail of edits in the window at once. It is the workload that exhausts a window fastest, and the one where compaction stops being optional — a two-hour session generates far more text than any window holds, however large.

**Prompt caching, and what it is worth.** Because the whole prefix is re-sent every turn, providers let you keep it warm: Anthropic bills cached prefix tokens at **0.1x** the normal input rate on most of its models — **0.025x** on Claude Fable 5.1 — and charges **1.25x** to write the cache. Put an agent with a 50,000-token codebase prefix through 100 turns in an hour, at $5 per million input tokens and the 0.1x read rate:

- **Without caching**: 100 x 50,000 = 5,000,000 input tokens → **$25 per hour**
- **With caching**: one write at 1.25x ($0.31) plus 99 reads at 0.1x ($2.48) → **$2.79 per hour**

Roughly **nine times cheaper**, for no change other than putting the stable content first and the variable content last — a prompt layout rule that falls straight out of the mechanism above.

**Document review.** Whole contracts, filings and research papers now fit in one request, removing the chunking machinery an earlier generation of tools needed. What it does not remove is the accuracy problem below — summarizing a document means combining facts scattered through it, which is precisely the workload long context is weakest at.

**Choosing between a big window and retrieval.** Pasting everything is simpler, costs more per request, and degrades as the pile grows. [Retrieval-augmented generation](https://howaiworks.ai/glossary/retrieval-augmented-generation) pulls in only the relevant passages and costs less, at the price of a retrieval stack you maintain. For one document that fits comfortably, use the window; for a corpus, retrieve — [RAG vs long context](https://howaiworks.ai/blog/best-llm-rag-long-context-summarization) covers how to read the benchmarks behind that call.

## Key Concepts

**Input budget and output budget are different numbers.** The advertised window is almost entirely *input*, and the cap on what the model can write back is an order of magnitude smaller:

| Model | Advertised context window | Max output |
|---|---|---|
| [Llama 4 Scout](https://howaiworks.ai/models/llama) (open weights) | 10,000,000 | — |
| [GPT-6 Astra](https://howaiworks.ai/models/gpt) | 1,050,000 (max input 922,000) | 128,000 |
| [Gemini 3.8 Flash](https://howaiworks.ai/models/gemini) | 1,048,576 | 65,536 |
| [Claude Fable 5.1](https://howaiworks.ai/models/claude-fable) | 1,000,000 | 128,000 |
| [Claude Opus 5](https://howaiworks.ai/models/claude-opus) | 1,000,000 | 128,000 |
| [Grok 4.3](https://howaiworks.ai/models/grok) | 1,000,000 | — |
| [Grok 4.6](https://howaiworks.ai/models/grok) | 500,000 | — |
| [Claude Haiku 4.5](https://howaiworks.ai/models/claude-haiku) | 200,000 | 64,000 |

GPT-6 Astra is the clearest illustration of why "context window" and "how much can I send" are two different questions. Its window is 1,050,000 tokens, but OpenAI caps input at **922,000** and output at **128,000** — and 922,000 + 128,000 = 1,050,000 exactly. The window is the *sum*, and the vendor has pre-divided it so that a maximum-length prompt still leaves room for a maximum-length answer. On the models that publish only one number, you do that division yourself: send 1,000,000 tokens to a 1M-token model and there is nothing left to write with.

These are advertised maximums, not effective ones. On reasoning models the internal thinking is spent from the output budget too, so a long chain of reasoning leaves less room for the visible answer.

**Context is not memory.** The window is per-conversation and vanishes when the chat ends. "Memory" in an assistant is a separate store that persists across sessions and injects a few relevant facts *back into* the window when needed. Different mechanisms, different expectations; [context window vs tokens vs memory](https://howaiworks.ai/blog/context-window-vs-tokens-vs-memory) untangles all three.

**Windows can shrink.** Grok 4.5 halved its predecessor's window, from 1,000,000 tokens down to 500,000, while raising the price — and [Grok 4.6](https://howaiworks.ai/models/grok), the current flagship, kept the smaller window rather than restoring the larger one. Grok 4.3 remains available specifically as the long-context option: 1,000,000 tokens at a lower per-token rate than either successor. Context length is a product decision with a cost behind it, not a capability ratchet that only moves up.

**Long context carries a price premium, and it arrives as a cliff rather than a slope.** This is the practical answer to "why not just always use the full window?" Two of the models above reprice the *entire* request once the prompt crosses a threshold — not the tokens above it, all of them:

- **[GPT-6 Astra](https://howaiworks.ai/models/gpt) steps at 272,000 input tokens.** Above that line the whole request bills at 2x the input and cache rates and 1.5x the output rate. Input alone: 272,000 tokens at $10 per million is **$2.72**; 273,000 tokens at $20 per million is **$5.46**. Those extra thousand tokens cost $2.74 — a marginal rate of roughly **$2,740 per million**, about 274 times the sticker price.
- **[Grok 4.6](https://howaiworks.ai/models/grok) steps at 200,000 tokens**, and *at* the threshold rather than above it. Reach 200,000 and every input, cached and output token in the call bills at $4.00 / $1.00 / $12.00 per million instead of $2.00 / $0.50 / $6.00. The window is 500,000 tokens; the part of it you can use at the headline price is the first 200,000.

Google's Gemini 3.1 Pro Preview is tiered the same way, at $2.00 in / $12.00 out per million up to 200K and $4.00 / $18.00 above it. Claude Opus 5 and Claude Fable 5.1 charge standard rates across their full 1M windows, with no threshold at all.

The consequence is a shape teams do not expect: an agent that grows its own context does not get gradually more expensive, it doubles in cost between one turn and the next, and the turn that crossed the line looks identical to the one before it. If you are working near a threshold, compact *before* it rather than after.

## Challenges

**Accuracy decays inside the window — "context rot".** As a window fills, the model's ability to reliably use what is in it drops, and the standard benchmark hides this. Single-needle "find the planted sentence" tests score above 90% at a million tokens for nearly every frontier model. Multi-needle tests — find eight facts in different places and combine them — fall off a cliff. Pick a model on the multi-needle numbers; the headline number tells you almost nothing about the work you will actually give it.

**Teams budget for the context and get billed for the turns.** A twenty-turn chat bills 10.5x its own length, so the surprise arrives as an *invoice*, not as an error. Caching is the fix, and it only works when the stable part of the prompt is byte-identical every time — a timestamp in the system prompt silently invalidates the entire prefix.

**Latency scales with prompt length, not answer length.** Before the first character appears the model must read the entire prompt — the prefill phase — and that work is proportional to how much you sent. A million-token request spends seconds reading before it emits anything, on every turn. A user waiting on a long-context request is waiting on tokens *they* wrote, not tokens the model is writing.

**More context can make the answer worse.** Irrelevant text in the window is not neutral: the model cannot tell your carefully chosen document from the four you added "just in case", and a bigger window is a standing invitation to stuff it. A smaller, curated context routinely beats a large noisy one — which is why [prompt engineering](https://howaiworks.ai/glossary/prompt-engineering) for long contexts is mostly about deciding what to leave out.

**Vendor numbers are not comparable.** 10M via interleaved positional encoding, 1M via a dense window, and 500K whose top 300K bills at double the headline rate are three products wearing the same unit. Nothing forces a lab to publish the length at which quality holds, and none of them volunteer it.

## Future Trends

**The competition is shifting from advertised size to effective length.** Once every frontier model claims about a million tokens, the number differentiates nothing, and the benchmarks that matter become the ones measuring whether the model can *use* the window — multi-fact retrieval and long-document summarization rather than single-needle recall.

**Context management is becoming an API primitive.** Compaction and context editing now ship as first-class API features. The direction of travel is that "what stays in the window" becomes a configurable policy the provider executes, not a summarization loop each team writes badly on its own.

**The cache is turning into a storage tier.** Prefix caches shared across users and requests, spilled to disk and pulled back when a conversation resumes, are already how large agent deployments avoid re-reading the same codebase a hundred times an hour. Once the cache is a durable, addressable object, the boundary between "context" and "memory" starts to dissolve — and the model's working set stops being something it rebuilds from nothing every time you say hello.

## Code Example

Measure before you send. Token counts are model-specific, so use the model's own counter rather than a generic tokenizer — and ask the API for the window instead of hardcoding a number that will change.

```python
from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-opus-5"

# The window is a published property of the model — look it up, don't hardcode it.
limit = client.models.retrieve(MODEL).max_input_tokens

conversation = [
    {"role": "user", "content": open("contract.txt").read()},
]

used = client.messages.count_tokens(model=MODEL, messages=conversation).input_tokens

print(f"{used:,} / {limit:,} tokens — {used / limit:.1%} of the window")

if used > limit * 0.4:
    print("Past the comfortable range. Retrieve the relevant passages instead of pasting everything.")
```

The 40% threshold is a judgement call, not a law — it reflects effective context sitting well below the advertised maximum. Move it based on how much your task depends on combining facts scattered through the text.

Note: this term was last reviewed in September 2026. The per-model figures cited — window sizes, output caps, per-token prices and the long-prompt billing thresholds — are stated as of that review and are the parts of this page most likely to age. The mechanisms around them, statelessness, the linear KV cache and the gap between advertised and effective length, are not.

## Frequently Asked Questions

### What is a context window in simple terms?

It is everything the model can see while it writes its next word — and the only thing it can see. The system instructions, every message so far, any document you pasted, and the reply being generated all share one fixed budget, measured in tokens. A model has no memory between turns; the window is not a summary of the conversation, it *is* the conversation.

### How much text fits in a 1 million token context window?

About 775,000 words of English, measured at 1.29 tokens per word across this site's own glossary prose with OpenAI's o200k_base encoder. That is seven or eight full-length novels, or a mid-sized codebase. It is also a real bill: at $5 per million input tokens, filling that window costs about $5 for a single request — and you pay it again on every turn unless the prefix is cached.

### Is a bigger context window always better?

No. A model's *effective* context — the length at which it still reasons reliably — is well below its advertised maximum. Frontier models advertise 1M tokens but hold up best under roughly 200–400K when a task requires combining several facts scattered through the text. Single-fact retrieval scores stay near-perfect at 1M, which is exactly why that benchmark misleads. The top of the window is also the expensive part: GPT-6 Astra reprices an entire request at 2x input once the prompt passes 272,000 tokens, and Grok 4.6 does the same at 200,000.

### Why does an AI chat forget what I said earlier?

Nothing was erased. The conversation grew past the window, so the oldest messages either fell outside what the model can see or were summarized to make room. An API call that exceeds the window returns an error instead; chat interfaces hide the limit by dropping history silently, which is what makes it feel like forgetting.

### Does the context window include the model's answer?

Yes, but the output has its own much smaller cap. Claude Opus 5 and Claude Fable 5.1 accept 1M input tokens and return at most 128,000; Gemini 3.8 Flash accepts 1,048,576 and returns 65,536. GPT-6 Astra separates the two explicitly: a 1,050,000-token context of which at most 922,000 may be input and 128,000 output. On reasoning models the internal thinking tokens are spent from that same output budget, so a long chain of reasoning leaves less room for the visible answer.

### What actually limits how large a context window can be?

Memory, not attention maths. Every token in the window leaves keys and values in the KV cache — for Llama 3.1 70B that is 320 KiB per token, so a single 1M-token session would hold about 328 GB. An eight-GPU H100 server has 640 GB total and spends 141 GB of it on the model's weights. A million-token window is a serving problem long before it is an architecture problem.

## Related

### Related terms

- [Token](https://howaiworks.ai/glossary/token)
- [KV Cache](https://howaiworks.ai/glossary/kv-cache)
- [Attention Mechanism](https://howaiworks.ai/glossary/attention-mechanism)
- [Large Language Model (LLM)](https://howaiworks.ai/glossary/large-language-model)
- [Retrieval-Augmented Generation (RAG)](https://howaiworks.ai/glossary/retrieval-augmented-generation)
- [Transformer](https://howaiworks.ai/glossary/transformer)

---

Source: https://howaiworks.ai/glossary/context-window — HowAIWorks.ai