Introduction
Three words get used interchangeably in AI conversations, and they mean three different things: tokens, context window, and memory. The confusion is not harmless — it's why people are surprised when a chatbot "forgets," frustrated when a bigger context window doesn't help, and wrong about what an assistant actually retains between sessions. Get the three straight and most of the mystery about why an AI chat behaves the way it does disappears.
This explainer defines each one plainly, shows how they fit together, and — the practical payoff — tells you which one is actually limiting your conversation at any given moment. No math beyond counting.
Tokens: the unit everything is measured in
A token is the basic unit of text a language model reads and writes. It's not quite a word and not quite a character — it's a chunk produced by tokenization, and a useful rule of thumb is that one token is about ¾ of a word in English (so ~750 words ≈ 1,000 tokens). Punctuation, spaces, and word pieces all get their own tokens.
Tokens are the currency of AI. You're billed per token, rate-limited per token, and — crucially — the model's capacity is measured in tokens. Every other limit in this article is ultimately counted in them. When a provider quotes a "1M context" or a price "per million tokens," this is the unit they mean.
The context window: the model's working memory
The context window is how many tokens the model can hold in view at once. It has to fit everything the model considers for its next response: your current prompt, the entire conversation so far, any documents you pasted, the system instructions, and the reply it's generating. If tokens are the currency, the context window is the size of the wallet.
This is the model's working memory — think of it as a whiteboard with a fixed size. Everything relevant has to be on the whiteboard at once, and when it fills up, something has to come off. Modern models have large whiteboards: as of 2026, DeepSeek V4, Claude Sonnet 5, GPT-5.5, and Gemini 3.5 all advertise around a million tokens, and Llama 4 Scout goes higher. But "large" is not "unlimited," and the advertised size is not the whole story.
Why models "forget" — and why bigger isn't always better
When a chat "forgets" what you said an hour ago, nothing was erased from a memory bank. The earlier text simply scrolled off the whiteboard: as the conversation grew past the window, the oldest messages fell outside what the model can see, or got summarized to make room. This is a context-window limit, and it's the single most common source of "the AI forgot" complaints. (Tools like Claude Code handle this explicitly — see managing auto-compaction.)
There's a subtler limit too: effective context. A model's usable context — the length at which it still reasons reliably — is well below its advertised maximum. Many 2026 models claim 1M tokens but perform best under roughly 200-400K for tasks that require combining several facts. Beyond that, quality quietly degrades even though the text technically "fit." So a bigger window is not automatically better; a smaller, well-curated context often beats a giant, noisy one. The whiteboard being large doesn't mean the model reads all of it equally well.
"Memory": persistence across conversations
Here's the one people conflate most. Memory, in the assistant sense, is persistence across separate conversations — facts the assistant deliberately saves and recalls in a future chat, like your name, your preferences, or a project's details. This is a completely different mechanism from the context window:
- The context window is temporary and lives inside one conversation. Close the chat and it's gone.
- Memory is durable and spans sessions. It's implemented as a separate store the assistant writes to and reads from, injecting relevant facts back into the context window when needed.
So when an assistant "remembers" your name next week, it didn't keep a week-long conversation in its window. It saved a small fact to memory and reloaded it. Persistent-memory systems — see our coverage of persistent memory agents — are a design layer on top of the model, not a property of the context window itself.
How they fit together
The clean mental model:
- Tokens are the unit. Everything is counted in them.
- Context window is how many tokens the model can see right now, in this conversation. Finite, temporary, and the usual cause of "forgetting."
- Memory is what persists between conversations. A separate store that feeds facts back into the window.
A useful analogy: tokens are words, the context window is the page you're currently reading, and memory is the notebook where you jot down things to remember for later. The page has a fixed size; the notebook persists.
What this means in practice
- Chat "forgot" something from earlier? That's the context window filling up. Start a fresh chat for a new topic, or restate the key facts.
- Pasting a huge document and getting vague answers? You may be past the effective context. Use retrieval-augmented generation to pull in only the relevant passages instead of everything.
- Want the assistant to remember you next time? That needs a memory feature, not a bigger context window — they're different tools for different jobs.
- Watching your bill? Every token in the window is paid for on every message, so a bloated context costs money as well as accuracy.
Conclusion
Tokens, context window, and memory are three layers, not three names for one thing. Tokens are the unit; the context window is the finite, per-conversation working memory that causes most "forgetting"; and memory is the durable store that persists across sessions. When an AI chat behaves in a way that surprises you, the fix usually starts with identifying which of the three you're actually bumping into — and more often than not, it's the context window. For the deeper mechanics, see our glossary on the context window, tokens, and the KV cache that stores attention state as the window fills.