Definition
Agent memory is how an AI agent keeps the information it needs within reach while it works — what it stores, where it stores it, and how it decides what to load back in. An agent built on a large language model has no memory of its own between calls: the model is stateless, and everything it "knows" on any given step is whatever text sits in its context window at that moment. Agent memory is the layer around the model that manages that scarce window — holding the recent conversation, writing older material to external storage, and pulling relevant facts back in on demand.
The distinction that matters most is short-term versus long-term. Short-term (working) memory is the live transcript inside the context window: the model can read all of it directly, but the window is finite, so it fills up. Long-term memory is everything kept outside the window in a database or file store — effectively unbounded, but invisible to the model until something retrieves it. A useful agent needs both, plus a policy for moving information between them, because the failure this whole apparatus exists to prevent is concrete: a long task generates more text than the window holds, and without management the agent either errors out or silently forgets the very instruction it was given at the start.
How It Works
Agent memory runs as a loop around each model call: store, retrieve, summarize, evict. On every step the agent assembles a prompt out of a system message, its tool definitions, some retrieved long-term facts, and the recent transcript, then calls the model. The model's reply — reasoning, a tool call, a tool result — is appended to the transcript. That transcript is the short-term memory, and it grows with every step. The job of the memory system is to keep that growing transcript inside the window while losing as little useful information as possible.
The clearest way to see the pressure is to count tokens. Suppose an agent runs on a model with a 128,000-token window (a common size as of 2026). Reserve 8,000 tokens for the system prompt and tool schemas that must be present on every call, and 4,000 for the model's own reply. That leaves a 116,000-token budget for conversation history. Now say each step of the task appends roughly 7,000 tokens — a 6,000-token tool result (a fetched web page, a file, a database dump) plus about 1,000 tokens of the model's reasoning and messages. Then:
116,000 / 7,000 ≈ 16 steps
The agent overflows the window after about 16 steps. On a real task — debugging across a dozen files, researching a question across many sources — sixteen steps is nothing. This is why "just use a bigger window" does not solve the problem: a longer task always exists, and a full window also costs more and slows down, because attention cost grows with the number of tokens in play.
Summarization is what buys the extra room. When the transcript nears the limit, the agent takes the oldest span — say the first 8 steps, about 56,000 tokens — and replaces it with a compact summary of what happened: the decisions made, the facts learned, the current state. If that summary comes to 700 tokens, the agent reclaims:
56,000 − 700 = 55,300 tokens (an 80x compression on that span)
Those reclaimed tokens buy roughly 55,300 / 7,000 ≈ 8 more steps before the next compaction, and the process repeats. The system that popularized this design, MemGPT (Packer et al., 2023, later the Letta project), makes the analogy to an operating system explicit: it treats the context window as main context — fast, small, like RAM — and an external database as external context — slow, large, like disk — and pages information between them. Its main context is split into read-only system instructions, a small read/write working context of pinned facts (user preferences, persona, key state), and a FIFO queue of recent messages whose first entry is "a recursive summary of messages that have been evicted from the queue." When the queue fills, MemGPT "generates a new recursive summary using the existing recursive summary and evicted messages" — exactly the summarize-and-evict step above, run automatically.
Long-term memory closes the loop. What gets evicted is not thrown away: the full detail is written to external storage — MemGPT calls its two forms recall storage (the complete message history) and archival storage (arbitrary text the agent chooses to save). On a later step, when the agent needs something it summarized away, it queries that store — usually by embedding the current situation and running a vector search for the most similar saved items — and pulls the top few back into the window. This is the same retrieval machinery as retrieval-augmented generation, pointed inward at the agent's own history instead of outward at a knowledge base.
One clarification worth making, because the words overlap: agent memory is not the KV cache. The KV cache is a compute optimization for tokens the model is already attending to; agent memory decides which tokens are in the window in the first place. Deciding what belongs in the window at all is the harder, higher-level problem, and it overlaps heavily with context engineering — the practice of curating exactly what a model sees on each call.
Real-World Applications
These are named, in-production uses of agent memory as of 2026:
- MemGPT / Letta — the OS-inspired tiered memory described above, now an open framework. Its recursive-summary-plus-external-store design is the reference implementation most later systems echo, and it is the clearest worked example of the store/retrieve/summarize/evict loop.
- ChatGPT Memory (OpenAI) — a consumer-facing long-term memory that persists facts about a user (name, preferences, ongoing projects) across separate conversations, so a new chat can start already knowing context the user never repeats. This is cross-session long-term memory in a shipping product.
- Coding agents — assistants that run long sessions, such as Claude Code and Cursor, summarize or "compact" the session transcript when it approaches the window, so a multi-hour editing session does not hit a hard wall mid-task. The compaction step is short-term memory management made visible to the user.
- Agent frameworks — LangGraph, CrewAI, and similar toolkits ship memory primitives: a short-term checkpointer that holds the running state and a long-term store keyed by user or thread, so builders get the store/retrieve loop without writing it from scratch.
The common thread is a task that outlives a single model call: a conversation across days, a project across sessions, a job across dozens of tool calls. Where a task fits inside one window in one sitting, an agent needs no memory system at all — and adding one is pure overhead.
Key Concepts
Working context is precious, external context is cheap. The whole design follows from that asymmetry. Every token in the window costs money and attention on every subsequent call, so the memory system's real job is curation: keep the few things the model needs this step, and demote everything else to storage it can query. A well-designed agent memory is less about remembering more and more about forgetting well — evicting the right things and keeping a faithful trace of what it evicted.
Shared versus isolated memory is the sharpest trade-off in multi-agent systems. When several agents work together, they either read and write one shared memory or each keep their own. A shared store keeps everyone aligned — the coder sees what the planner decided — but it grows large, costs more tokens for every agent that must load it, and lets one agent's bad write mislead all the others. Isolated stores keep each agent focused and cheap, but agents then act on partial pictures and produce work that does not fit together, the failure mode of context fragmentation. There is no universally right answer; the choice depends on how tightly the agents' subtasks are coupled. This is the tension that sits behind the observation that getting context sharing wrong is the root of most multi-agent failures.
Memory is state, and state can be wrong. A fact written to long-term memory in March ("the user prefers TypeScript") may be false by July, and unless something invalidates it, the agent will keep acting on it. Long-term memory turns an agent from stateless-and-forgetful into stateful-and-possibly-stale, which is a different and sometimes harder problem.
Challenges
The hardest problems in agent memory come from the fact that both compression and retrieval are lossy, and the loss is invisible until it bites.
- Summarization silently drops the load-bearing detail. A summary keeps what looked important when it was written, but importance is only clear later. An agent told a specific constraint on turn 3 — "never modify the billing table" — can have that line compressed into a bland "discussed database safety" by turn 30, and then cheerfully violate it, because the exact wording is gone. Recursive summaries make this worse: a summary of a summary of a summary drifts, each pass losing fidelity the earlier ones preserved.
- Retrieval misses are unobservable. Long-term memory only helps if the right item is retrieved at the right moment. Vector search returns the top k results; a fact that ranks k+1 is functionally absent, and the agent has no way to know a relevant memory existed. Unlike an overflow error, a retrieval miss produces no signal — the agent just proceeds without the fact.
- Memory can be poisoned. Because tool results and web pages flow into memory, a malicious or wrong item written to long-term storage persists and re-injects itself on every later retrieval that matches it. This turns a one-time prompt injection into a durable one, embedded in the agent's own memory.
- Coordination cost in the shared case. When agents share memory, every write is a potential read for every other agent, and keeping that store consistent adds latency and token cost that scales with the number of agents — part of why multi-agent systems are expensive to run.
Future Trends
- Memory as a managed layer. Rather than each team hand-rolling a store/retrieve/summarize loop, dedicated memory services (Letta, mem0, and similar) are emerging to provide it as infrastructure — the direction of travel is memory becoming a component you plug in, not code you write.
- Learned eviction and retrieval policies. Today's systems mostly evict by recency and retrieve by embedding similarity, both crude heuristics. The open research direction is agents that learn what to remember and when to recall it based on the task, rather than following a fixed rule.
- Standardized memory across agents. As agent-to-agent protocols mature, the same standardization that connected agents to tools is beginning to reach memory — letting agents built by different teams share a memory substrate instead of each keeping a private, incompatible store.