--- source: 'https://howaiworks.ai/glossary/agent-memory' section: glossary title: Agent Memory description: >- Agent memory is how an AI agent stores, retrieves, and summarizes information so it works across long tasks and sessions without overflowing its context window. tags: - AI Agents - LLM - Context Window - Long Context - RAG - information retrieval category: AI Applications datePublished: '2026-07-24' lastUpdated: '2026-07-24' --- # Agent Memory > Agent memory is how an AI agent stores, retrieves, and summarizes information so it works across long tasks and sessions without overflowing its context window. ## Definition **Agent memory** is how an AI agent keeps the information it needs within reach while it works — what it stores, where it stores it, and how it decides what to load back in. An agent built on a [large language model](https://howaiworks.ai/glossary/large-language-model) has no memory of its own between calls: the model is stateless, and everything it "knows" on any given step is whatever text sits in its [context window](https://howaiworks.ai/glossary/context-window) at that moment. Agent memory is the layer around the model that manages that scarce window — holding the recent conversation, writing older material to external storage, and pulling relevant facts back in on demand. The distinction that matters most is short-term versus long-term. **Short-term (working) memory** is the live transcript inside the context window: the model can read all of it directly, but the window is finite, so it fills up. **Long-term memory** is everything kept outside the window in a database or file store — effectively unbounded, but invisible to the model until something retrieves it. A useful agent needs both, plus a policy for moving information between them, because the failure this whole apparatus exists to prevent is concrete: a long task generates more text than the window holds, and without management the agent either errors out or silently forgets the very instruction it was given at the start. ## How It Works Agent memory runs as a loop around each model call: **store, retrieve, summarize, evict.** On every step the agent assembles a prompt out of a system message, its tool definitions, some retrieved long-term facts, and the recent transcript, then calls the model. The model's reply — reasoning, a tool call, a tool result — is appended to the transcript. That transcript is the short-term memory, and it grows with every step. The job of the memory system is to keep that growing transcript inside the window while losing as little useful information as possible. The clearest way to see the pressure is to count tokens. Suppose an agent runs on a model with a **128,000-token** window (a common size as of 2026). Reserve **8,000** tokens for the system prompt and tool schemas that must be present on every call, and **4,000** for the model's own reply. That leaves a **116,000-token** budget for conversation history. Now say each step of the task appends roughly **7,000** tokens — a 6,000-token tool result (a fetched web page, a file, a database dump) plus about 1,000 tokens of the model's reasoning and messages. Then: 116,000 / 7,000 ≈ 16 steps The agent overflows the window after about **16 steps**. On a real task — debugging across a dozen files, researching a question across many sources — sixteen steps is nothing. This is why "just use a bigger window" does not solve the problem: a longer task always exists, and a full window also costs more and slows down, because attention cost grows with the number of tokens in play. **Summarization** is what buys the extra room. When the transcript nears the limit, the agent takes the oldest span — say the first **8 steps**, about **56,000** tokens — and replaces it with a compact summary of what happened: the decisions made, the facts learned, the current state. If that summary comes to **700** tokens, the agent reclaims: 56,000 − 700 = 55,300 tokens (an 80x compression on that span) Those reclaimed tokens buy roughly `55,300 / 7,000 ≈ 8` more steps before the next compaction, and the process repeats. The system that popularized this design, **MemGPT** (Packer et al., 2023, later the Letta project), makes the analogy to an operating system explicit: it treats the context window as **main context** — fast, small, like RAM — and an external database as **external context** — slow, large, like disk — and pages information between them. Its main context is split into read-only system instructions, a small read/write **working context** of pinned facts (user preferences, persona, key state), and a **FIFO queue** of recent messages whose first entry is *"a recursive summary of messages that have been evicted from the queue."* When the queue fills, MemGPT "generates a new recursive summary using the existing recursive summary and evicted messages" — exactly the summarize-and-evict step above, run automatically. Long-term memory closes the loop. What gets evicted is not thrown away: the full detail is written to external storage — MemGPT calls its two forms **recall storage** (the complete message history) and **archival storage** (arbitrary text the agent chooses to save). On a later step, when the agent needs something it summarized away, it queries that store — usually by [embedding](https://howaiworks.ai/glossary/embedding) the current situation and running a [vector search](https://howaiworks.ai/glossary/vector-search) for the most similar saved items — and pulls the top few back into the window. This is the same retrieval machinery as [retrieval-augmented generation](https://howaiworks.ai/glossary/retrieval-augmented-generation), pointed inward at the agent's own history instead of outward at a knowledge base. One clarification worth making, because the words overlap: agent memory is **not** the [KV cache](https://howaiworks.ai/glossary/kv-cache). The KV cache is a compute optimization for tokens the model is *already* attending to; agent memory decides which tokens are in the window in the first place. Deciding what belongs in the window at all is the harder, higher-level problem, and it overlaps heavily with [context engineering](https://howaiworks.ai/glossary/context-engineering) — the practice of curating exactly what a model sees on each call. ## Real-World Applications These are named, in-production uses of agent memory as of 2026: - **MemGPT / Letta** — the OS-inspired tiered memory described above, now an open framework. Its recursive-summary-plus-external-store design is the reference implementation most later systems echo, and it is the clearest worked example of the store/retrieve/summarize/evict loop. - **ChatGPT Memory (OpenAI)** — a consumer-facing long-term memory that persists facts about a user (name, preferences, ongoing projects) across separate conversations, so a new chat can start already knowing context the user never repeats. This is cross-session long-term memory in a shipping product. - **Coding agents** — assistants that run long sessions, such as Claude Code and Cursor, summarize or "compact" the session transcript when it approaches the window, so a multi-hour editing session does not hit a hard wall mid-task. The compaction step is short-term memory management made visible to the user. - **Agent frameworks** — LangGraph, CrewAI, and similar toolkits ship memory primitives: a short-term checkpointer that holds the running state and a long-term store keyed by user or thread, so builders get the store/retrieve loop without writing it from scratch. The common thread is a task that outlives a single model call: a conversation across days, a project across sessions, a job across dozens of tool calls. Where a task fits inside one window in one sitting, an agent needs no memory system at all — and adding one is pure overhead. ## Key Concepts **Working context is precious, external context is cheap.** The whole design follows from that asymmetry. Every token in the window costs money and attention on *every* subsequent call, so the memory system's real job is curation: keep the few things the model needs this step, and demote everything else to storage it can query. A well-designed agent memory is less about remembering more and more about forgetting well — evicting the right things and keeping a faithful trace of what it evicted. **Shared versus isolated memory is the sharpest trade-off in [multi-agent systems](https://howaiworks.ai/glossary/multi-agent-systems).** When several agents work together, they either read and write one shared memory or each keep their own. A **shared** store keeps everyone aligned — the coder sees what the planner decided — but it grows large, costs more tokens for every agent that must load it, and lets one agent's bad write mislead all the others. **Isolated** stores keep each agent focused and cheap, but agents then act on partial pictures and produce work that does not fit together, the failure mode of context fragmentation. There is no universally right answer; the choice depends on how tightly the agents' subtasks are coupled. This is the tension that sits behind the observation that getting context sharing wrong is the root of most multi-agent failures. **Memory is state, and state can be wrong.** A fact written to long-term memory in March ("the user prefers TypeScript") may be false by July, and unless something invalidates it, the agent will keep acting on it. Long-term memory turns an agent from stateless-and-forgetful into stateful-and-possibly-stale, which is a different and sometimes harder problem. ## Challenges The hardest problems in agent memory come from the fact that both compression and retrieval are lossy, and the loss is invisible until it bites. - **Summarization silently drops the load-bearing detail.** A summary keeps what looked important when it was written, but importance is only clear later. An agent told a specific constraint on turn 3 — "never modify the billing table" — can have that line compressed into a bland "discussed database safety" by turn 30, and then cheerfully violate it, because the exact wording is gone. Recursive summaries make this worse: a summary of a summary of a summary drifts, each pass losing fidelity the earlier ones preserved. - **Retrieval misses are unobservable.** Long-term memory only helps if the right item is retrieved at the right moment. Vector search returns the top *k* results; a fact that ranks *k*+1 is functionally absent, and the agent has no way to know a relevant memory existed. Unlike an overflow error, a retrieval miss produces no signal — the agent just proceeds without the fact. - **Memory can be poisoned.** Because tool results and web pages flow into memory, a malicious or wrong item written to long-term storage persists and re-injects itself on every later retrieval that matches it. This turns a one-time [prompt injection](https://howaiworks.ai/glossary/prompt-injection) into a durable one, embedded in the agent's own memory. - **Coordination cost in the shared case.** When agents share memory, every write is a potential read for every other agent, and keeping that store consistent adds latency and token cost that scales with the number of agents — part of why multi-agent systems are expensive to run. ## Future Trends - **Memory as a managed layer.** Rather than each team hand-rolling a store/retrieve/summarize loop, dedicated memory services (Letta, mem0, and similar) are emerging to provide it as infrastructure — the direction of travel is memory becoming a component you plug in, not code you write. - **Learned eviction and retrieval policies.** Today's systems mostly evict by recency and retrieve by embedding similarity, both crude heuristics. The open research direction is agents that learn *what* to remember and *when* to recall it based on the task, rather than following a fixed rule. - **Standardized memory across agents.** As agent-to-agent protocols mature, the same standardization that connected agents to tools is beginning to reach memory — letting agents built by different teams share a memory substrate instead of each keeping a private, incompatible store. ## Frequently Asked Questions ### What is agent memory? Agent memory is the machinery an AI agent uses to keep the information it needs available while it works — holding recent turns in the model's context window, storing older facts in an external database, and summarizing or evicting material so the window never overflows. ### What is the difference between short-term and long-term agent memory? Short-term (working) memory is whatever currently sits inside the model's context window — the live transcript the model can read on this step. Long-term memory is information kept outside the window in a store, retrieved back in only when it becomes relevant. Short-term is fast but capped by the window; long-term is effectively unbounded but only useful if the right item is retrieved. ### How do agents avoid running out of context? They budget the window. When the running transcript approaches the limit, the agent summarizes or evicts older turns — replacing, say, 56,000 tokens of past tool output with a 700-token summary — and pushes the full detail to an external store it can query later. ### Is agent memory the same as RAG? They share machinery but not purpose. Retrieval-augmented generation pulls facts from an external knowledge base to answer a question; agent memory stores and retrieves the agent's own history and state so it stays coherent across a long task. Long-term agent memory is usually implemented with the same retrieval tools RAG uses. ### Should agents in a multi-agent system share memory? It is a genuine trade-off. Shared memory keeps agents aligned but grows large, costs more tokens per agent, and lets one agent's bad write mislead the others. Isolated memory keeps each agent focused and cheap but risks conflicting assumptions and duplicated work. ## Related ### Related terms - [AI Agent](https://howaiworks.ai/glossary/ai-agent) - [Multi-Agent Systems (MAS)](https://howaiworks.ai/glossary/multi-agent-systems) - [Context Window](https://howaiworks.ai/glossary/context-window) - [Context Engineering](https://howaiworks.ai/glossary/context-engineering) - [Retrieval-Augmented Generation (RAG)](https://howaiworks.ai/glossary/retrieval-augmented-generation) - [Agentic Workflow](https://howaiworks.ai/glossary/agentic-workflow) --- Source: https://howaiworks.ai/glossary/agent-memory — HowAIWorks.ai