Context Engineering

Definition

Context engineering is the practice of deciding which tokens enter a model's context window on each request — and which ones get evicted to make room. Prompt engineering is one part of it. Prompt engineering writes the instructions; context engineering governs everything else that lands in the window beside them: tool schemas, retrieved documents, the output of every tool the model has called, and a conversation history that grows with every step.

The operational difference is that a prompt is authored once, while a context is assembled fresh on every single request. Anthropic's engineering team, whose 29 September 2025 post put the term into wide circulation, draws the line precisely: prompt engineering is "methods for writing and organizing LLM instructions for optimal outcomes", while context engineering is "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference, including all the other information that may land there outside of the prompts". One is authorship. The other is an allocation you re-make dozens of times inside a single task.

Here is the fact that turns this from housekeeping into a discipline. In Lost in the Middle, a study by researchers at Stanford, UC Berkeley and Samaya AI, GPT-3.5-Turbo was measured answering questions over a set of documents. Asked with no documents at all, it scored 56.1%. Given 20 or 30 documents including the one containing the answer, but with that document positioned in the middle, it scored lower than 56.1%. The correct information was in the window, and the model did worse than if it had never been shown anything. More context is not monotonically better, which means somebody has to decide what goes in — and that decision is the job.

How It Works

Five things compete for one budget

A request to a language model is one flat sequence of tokens. Nothing in it is privileged; the model does not know which part you consider important. Five kinds of content contend for that space, and they behave very differently.

The system prompt is small, stable and shapes everything — the highest-leverage tokens you own, and typically the smallest slice. Tool definitions are the names, descriptions and JSON schemas of everything the agent may call; they are large, they are re-sent verbatim on every request, and they arrive before the user has said a word. Retrieved content is whatever you pulled in for this particular question — the only component that is genuinely about the task. Message history is the accumulating transcript of reasoning, tool calls and tool results, and it is the only component that grows without bound. Finally, the answer is spent from the same window, which is easy to forget until a long prompt truncates the reply.

The reason these can be traded against each other at all is that the model is stateless. Turn 30 is a fresh request containing turns 1 through 29 verbatim — the mechanism the context window page works through in detail. Every token you leave in is a token you pay for again on the next step, and the step after that.

A worked budget for one agent turn

Take a realistic coding agent: a 200,000-token working budget, connected to five MCP servers, doing retrieval over a document store. The tool figure is not invented — Anthropic's published breakdown for a five-server setup (GitHub 35 tools at ~26K tokens, Slack 11 at ~21K, Sentry 5 at ~3K, Grafana 5 at ~3K, Splunk 2 at ~2K) totals 58 tools consuming approximately 55,000 tokens before any work begins.

Read the second row again. Over a quarter of the budget is gone before the user types anything, spent entirely on a menu of things the agent probably will not call. The system prompt everyone spends a week tuning is 1%.

Now run the loop. A single agent step — a paragraph of reasoning, a tool call, and the tool's result — costs on the order of 1,500 tokens. Dividing 119,000 by 1,500 gives 79 steps before the window is full. That is an afternoon of work on one bug, not a career, and it is why compaction exists.

The bill is worse than the budget suggests, because statelessness re-sends the prefix. The fixed 81,000 tokens are billed on all 79 steps, and the history is billed on average at half its final size:

fixed:    81,000 x 79                    =  6,399,000
history:  1,500 x (1 + 2 + ... + 79)     =  4,740,000
                                            ----------
                                            11,139,000 input tokens

Eleven million input tokens for one task — about $56 at $5 per million — of which 55,000 × 79 = 4,345,000 tokens, nearly 40% of the whole bill, is the tool menu being re-read on every step.

That last observation is where the leverage is. Anthropic's Tool Search Tool defers schema loading, and both its engineering write-up and the developer documentation put the reduction at over 85%. Applying that to the 55,000-token block leaves about 8,250 tokens and frees 46,750:

• history budget rises from 119,000 to 165,750 tokens — 79 steps become 110, a 39% longer run before compaction;
• over those first 79 steps it removes 3,693,250 billed input tokens, roughly $18 of the $56.

No prompt was rewritten. No model was changed. The gain came entirely from deciding that 55 tool schemas did not need to be in the room.

What retrieval is allowed to cost

Retrieval is the component most likely to be over-provisioned, because "return the top k" makes k look free. It is not, and the ceiling on what more retrieval buys arrives early. The Lost in the Middle authors ran a standard retriever-reader setup over NaturalQuestions-Open and measured reader accuracy against k. Their finding: "reader model performance saturates long before retriever performance saturates" — going beyond 20 retrieved documents improved the reader by roughly 1.5% for GPT-3.5-Turbo and about 1% for Claude-1.3, "while significantly increasing the input context length (and thus latency and cost)".

So the retrieval budget has a shape. Below some k you are losing answers to missing evidence; above it you are paying full price — on every subsequent turn, because the chunks stay in the history — for one point of accuracy. The engineering response the paper itself proposes is reranking and ranked-list truncation: fewer, better-ordered passages rather than more of them. RAG tutorials that set k=50 because the window can hold it are optimising the wrong variable.

The alternative to pre-loading is loading on demand. Anthropic calls this "just in time" context: keep lightweight identifiers — file paths, stored queries, links — and use tools to pull the content in only when it is needed. Claude Code works this way, dropping CLAUDE.md into context up front while using glob and grep to reach everything else, which "effectively bypass[es] the issues of stale indexing". The trade is explicit in the same post: "runtime exploration is slower than retrieving pre-computed data".

Order matters, because position does

Once you have decided what goes in, you still have to decide where. Models do not read a window uniformly. Lost in the Middle found a U-shaped curve: accuracy is highest when the relevant passage sits at the very beginning (primacy) or the very end (recency) of the context, and "performance significantly degrades when models must access and use information in the middle". GPT-3.5-Turbo's multi-document accuracy "can drop by more than 20%" purely from moving the answer's position, with the model unchanged and the text identical.

The effect is not universal across sizes, which is itself a useful detail: the paper reports that the 7B Llama-2 model was solely recency-biased, while the 13B and 70B models showed the full U-shape. Primacy appears to be something larger models acquire.

Three layout rules fall straight out of this. Put stable, high-value content — instructions, the task definition — at the front. Put the immediate question last. And never let a reranker leave the single best passage in position 10 of 20, which is the default behaviour of a system that sorts by score and then prints in order.

The first rule pays twice, because prefix caching also rewards a stable front: an unchanging prefix can be re-read from cache at a fraction of the input price, while a timestamp near the top invalidates everything after it. Layout for attention and layout for the KV cache happen to want the same thing.

Eviction: what leaves, and how

Every agent that runs long enough hits the wall computed above. There are three established ways to get past it, and they trade differently.

Compaction summarizes the conversation and restarts from the summary. Anthropic describes Claude Code's implementation as preserving "architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs or messages", then continuing "with this compressed context plus the five most recently accessed files". The recipe is more interesting than the idea: what survives is decisions and open problems, and what dies is raw tool output, because a tool result is evidence for a conclusion you have already drawn. The lightest-touch version is tool-result clearing on its own — dropping the bodies of old tool responses while keeping the record that they were called. Anthropic's own guidance for tuning a compaction prompt is to "start by maximizing recall... then iterate to improve precision", in that order, because dropping something critical is much more expensive than keeping something redundant.

Structured note-taking moves state out of the window entirely, into a file the agent writes and re-reads. This is the cheapest form of persistence, and unlike compaction it survives a full context reset rather than degrading through one.

Sub-agents partition the window rather than compressing it. A sub-agent explores with "tens of thousands of tokens or more" in its own clean context and returns "only a condensed, distilled summary of its work (often 1,000-2,000 tokens)". The compression ratio is the point: 40,000 tokens of searching becomes 1,500 tokens in the coordinator's window. What the coordinator loses is the ability to check the working, which is exactly why this suits research and search and suits debugging much less well. See multi-agent systems for the coordination side of that trade.

Real-World Applications

Claude Code's auto-compaction. The clearest shipped example: a coding agent that summarizes its own transcript when it nears the limit and continues from the summary plus its five most recent files, so a multi-hour session outlives a window that could never have held it. How that behaves in practice is a user-visible feature, not an implementation detail — the agent tells you it is compacting.

Deferred tool loading in the Claude API. Tool Search Tool exists specifically because connecting five MCP servers costs ~55,000 tokens of schema. Anthropic's internal MCP evaluations put the accuracy effect at 49% to 74% for Claude Opus 4 and 79.5% to 88.1% for Opus 4.5 with tool search enabled. That is not only a token saving: showing the model fewer options made it choose better, which is the attention-dilution mechanism described on the function calling page appearing as a measured number.

Multi-agent research systems. Anthropic's research agent uses the sub-agent pattern above — parallel exploration, condensed returns — and reports a substantial improvement over a single-agent system on complex research tasks. The context engineering is the architecture here; there is no separate design.

Long-horizon agents with external memory. The Claude Developer Platform ships a file-based memory tool for exactly this, letting an agent build a knowledge base across sessions instead of re-deriving it. Anthropic's illustration is Claude playing Pokémon, where the agent maintains tallies across thousands of game steps — "for the last 1,234 steps I've been training my Pokémon in Route 1, Pikachu has gained 8 levels toward the target of 10" — and resumes multi-hour sequences after a context reset by reading its own notes.

Key Concepts

Context rot. The observed effect that as a window fills, the model's ability to accurately recall what is in it declines. Anthropic states it as a property of every model tested: "While some models exhibit more gentle degradation than others, this characteristic emerges across all models." It is a gradient, not a cliff — but it means the effective capacity of a window is smaller than its stated capacity, and by an amount nobody publishes.

Attention budget. The useful mental model for why. A transformer lets every token attend to every other token, producing n² pairwise relationships for n tokens; as n grows, the model's capacity to represent those relationships is spread thinner. Doubling the context does not double the attention available — it quadruples the demands on it. This is also why the fix is curation rather than a bigger window.

Fixed cost versus growing cost. Two different problems that get conflated. Tool schemas and the system prompt are a constant re-billed once per step; the transcript is a growing block whose total cost rises with the square of the step count. They call for opposite remedies — you delete a constant once, and you must keep compacting a growing one.

Just-in-time versus pre-loaded. Whether context is assembled before inference (embeddings, retrieval, everything pasted up front) or pulled in during it (the agent calls a tool to fetch what it needs). Pre-loading is faster and stales; just-in-time is current and slower. Most production agents are hybrids, and choosing the boundary is a context-engineering decision.

Challenges

The measurement does not exist by default. Almost no agent framework tells you the composition of the request it just sent — you see a total token count, not the split between schemas, history and retrieval. Teams therefore optimise the part they can see (the prompt) and ignore the 27% they cannot. The first real intervention on most agents is printing the table above.

Compaction destroys the thing you needed, silently. Anthropic names this directly: "overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later". The failure has no error mode. The agent continues confidently, having forgotten the constraint it agreed to twenty steps ago, and the bug surfaces as a wrong decision rather than a missing-context exception.

Eviction and prefix caching are in direct conflict. Caching pays only for a byte-identical prefix. Deleting a stale tool result from the middle of the transcript invalidates the cache from that point to the end — so the turn on which you save 20,000 tokens of context is also the turn on which you re-pay full input price for everything after the deletion. This is why tool-result clearing and compaction are batched at a threshold instead of run continuously, and why context editing policies tend to remove content from the end of a stable prefix rather than the middle.

Multimodal content blows the budget without looking large. A screenshot occupies thousands of tokens, and unlike text it gives no visual cue of its size in a transcript — nothing about a thumbnail says "this cost more than the system prompt". In an agent loop each frame is re-sent on every subsequent step, so a multimodal agent that browses or drives a computer exhausts its window in a few dozen steps rather than a few hundred. Dropping stale frames from history is a requirement for those agents, not an optimisation.

Nobody agrees what "effective context" means, so you cannot shop for it. Vendors publish maximum window sizes. The number you need — the length at which your task still works — is task-specific, unpublished and must be measured on your own evaluation set. A model whose single-needle retrieval is perfect at 1M tokens may be unusable at 200K for a task requiring several facts combined.

Prompt-engineering habits transfer badly. The instinct built by chat interfaces is that more explanation helps: add another example, another edge case, another clarification. On a per-turn basis that is often true. Across a 79-step agent run, every clarification is a token re-billed 79 times and a competitor for attention on all of them. Anthropic's guidance inverts the habit — curate "a set of diverse, canonical examples" instead of stuffing "a laundry list of edge cases into a prompt".

Future Trends

Eviction is becoming a provider-side policy. Tool-result clearing and context editing already ship as API features rather than loops each team writes badly. The direction is that "what stays in the window" becomes a declarative policy the provider executes, in the same way garbage collection stopped being something application code did by hand.

Deferred tool loading is on its way to being the default. The current arrangement — send every schema on every request and hope the model reads the right one — looks increasingly like sending an entire database to the client because the client might query it. Both MCP-side tool search and code-execution approaches that let the model call tools programmatically point the same way: the tool catalog becomes something an agent queries, not something it carries.

The interesting benchmark moves from window size to eviction quality. Once every frontier model advertises around a million tokens, the differentiator is not how much fits but how well a system decides what to drop. Evaluations that resume an agent after compaction and check whether it still knows what it agreed to are the ones that will separate implementations, and they barely exist yet.

Context engineering is being absorbed into the model. Anthropic's own conclusion is that "smarter models require less prescriptive engineering". If a model can be trusted to manage its own working set — deciding when to summarise, what to fetch, what to forget — then much of what is described on this page becomes a learned behaviour rather than an application-layer scaffold. The budget does not go away; the question is who administers it.

Code Example

The intervention that matters most is the one nobody performs: count the components before sending. This script is the table above as arithmetic — replace the constants with counts from your model's own token counter, since token counts are model-specific.

# One agent turn, accounted honestly. Sizes are the measured ones cited above;
# replace them with your own token counts before trusting the totals.
WINDOW = 200_000        # the budget you choose to manage, not the model's maximum
OUTPUT_RESERVE = 8_000  # the answer is spent from the same window

fixed = {
    "system prompt": 2_000,
    "tool schemas (58 tools, 5 servers)": 55_000,
    "retrieved chunks (20 x 800)": 16_000,
    "output reserve": OUTPUT_RESERVE,
}

overhead = sum(fixed.values())
history = WINDOW - overhead
STEP = 1_500            # reasoning + tool call + tool result, per agent step
steps = history // STEP

for name, cost in fixed.items():
    print(f"{name:36} {cost:>7,}  {cost / WINDOW:5.1%}")
print(f"{'left for history':36} {history:>7,}  {history / WINDOW:5.1%}")
print(f"\n{steps} steps before compaction, at {STEP:,} tokens per step")

# Statelessness: every step re-sends the whole prefix.
billed = overhead * steps + STEP * steps * (steps + 1) // 2
print(f"{billed:,} input tokens billed for the task (${billed / 1e6 * 5:,.2f} at $5/M)")

# Now defer the tool definitions: Anthropic reports over 85% off the schema block.
deferred = int(55_000 * 0.15)
freed = 55_000 - deferred
print(f"\ndefer tool schemas: {55_000:,} -> {deferred:,}, freeing {freed:,} tokens")
print(f"history budget {history:,} -> {history + freed:,} = {(history + freed) // STEP} steps")
print(f"saved over the first {steps} steps: {freed * steps:,} tokens (${freed * steps / 1e6 * 5:,.2f})")

Running it prints:

system prompt                          2,000   1.0%
tool schemas (58 tools, 5 servers)    55,000  27.5%
retrieved chunks (20 x 800)           16,000   8.0%
output reserve                         8,000   4.0%
left for history                     119,000  59.5%

79 steps before compaction, at 1,500 tokens per step
11,139,000 input tokens billed for the task ($55.69 at $5/M)

defer tool schemas: 55,000 -> 8,250, freeing 46,750 tokens
history budget 119,000 -> 165,750 = 110 steps
saved over the first 79 steps: 3,693,250 tokens ($18.47)

The per-step figure of 1,500 tokens is the one to replace first: it varies enormously with how verbose your tools are, and a single tool that returns an unfiltered 10,000-token JSON blob will cut the step count by a factor of five on its own. Measure it, then decide whether the fix is compaction or a tool that returns less.

For the argument behind these techniques in full, Anthropic's effective context engineering for AI agents is the primary source; the mechanics of the window itself are on context window, and the loop that consumes it on AI agent.