Token

Definition

A token is one entry from a model's fixed vocabulary — most often a fragment of a word — and it is the only thing an AI model ever sees. Your text is converted into a list of integers before the model runs, the model does arithmetic on those integers, and integers come back out. The words you read were never in the building.

That conversion is not a detail, because everything is counted in tokens: what you pay, what fits in the context window, how much the model may write back, and how much GPU memory your session occupies while you hold it. Measured across 244,000 words of this site's own prose with OpenAI's o200k_base encoder, the exchange rate is 5.7 characters per token, or 1.21 tokens per English word — so 1,000 words costs roughly 1,200 tokens. The old rule of thumb that a token is four characters comes from GPT-3's tokenizer and understates modern encoders by a third.

This page is about the units and what they cost. Tokenization is about how the vocabulary that produces them gets built.

How It Works

A token id is an index. The model holds an embedding table with one row per vocabulary entry, so token 101830 means "fetch row 101830" — a vector of a few thousand numbers that is the model's learned representation of that fragment. Every layer transforms that vector in the context of its neighbours, and the final layer produces a score for every entry in the vocabulary. Sampling one of them yields the next token, which is appended to the sequence, and the whole thing runs again.

That loop is the reason output costs more than input. A prompt of any length is processed in a single pass, because all its tokens are known at once and the arithmetic parallelises across them. Output has no such luxury: token 400 cannot be computed until token 399 exists, so a 500-token reply means 500 sequential passes through every layer of the model. The pricing reflects the physics — GPT-5.6 charges $5 per million input tokens against $30 per million output, Claude Opus 4.8 $5 against $25, Gemini 3.6 Flash $1.50 against $7.50. Every frontier vendor lands on roughly the same 5:1 to 6:1 ratio because they are all paying for the same asymmetry.

Tokens also occupy memory for as long as you keep them. Each token already processed leaves behind cached key and value vectors so it does not have to be recomputed — the KV cache. For Llama 3.1 70B that costs 320 KiB per token, so a single user at a 128K context is holding 42.9 GB of GPU memory, more than half an H100. A token is not a character with a price tag; it is a character with a price tag and a rent.

Types

The categories below are not a conceptual scheme — they are the line items on an API invoice, and mistaking one for another is how estimates go wrong by an order of magnitude.

Input tokens

Everything you send: system prompt, conversation history, retrieved documents, tool definitions and tool results. In a multi-turn conversation the entire history is resent on every request, so input grows quadratically with turn count even though each individual message is small.

Output tokens

Everything the model generates, billed at 5-6x the input rate. Providers cap them separately from the context window: GPT-5.6 accepts 1,050,000 tokens in but will emit at most 128,000.

Reasoning tokens

Tokens a reasoning model produces for itself before answering. They are billed as output, they consume the output cap — on GPT-5.6 the same 128,000 — and they are not returned to you. A request can therefore fail to produce a visible answer while having been charged in full, which is a failure mode that did not exist before reasoning models.

Cached tokens

Input the provider has already processed and can replay cheaply. GPT-5.6 bills cached input at $0.50 per million against $5.00 — a 10x discount — provided the prefix is byte-identical. Gemini 3.5 prices it as $0.15 per million plus $1.00 per hour of storage, a rental model rather than a discount.

Special and control tokens

Vocabulary entries that mean something structural rather than lexical. BERT uses [CLS], [SEP], [MASK] and [PAD]; chat models use control tokens such as <|im_start|> to mark where the system prompt ends and the user's turn begins. This is why a chat request is not simply your strings concatenated: the template inserts control tokens, and they are billed like any other.

Tokens that are not text

Images, audio and video are converted to tokens too, and billed as input. The count tracks resolution — Claude Sonnet 5 raised its limit from 1568 to 2576 pixels on the long edge, and a high-resolution image can consume roughly 3x more image tokens as a result. A screenshot pasted into a prompt is not free context.

Real-World Applications

Which side of the invoice dominates depends entirely on the workload, and it flips. Take GPT-5.6 at $5 in and $30 out per million, and two jobs of similar size:

Same model, same price list, opposite conclusion about what to optimise. Shortening prompts is wasted effort on the second job; shortening replies is wasted effort on the first. You cannot know which you are running without counting both sides.

Caching changes the arithmetic more than model choice does. Ask twenty questions about the same 50,000-token document and you send a million input tokens: $5.00 at full rate. With the prefix cached at $0.50 per million, the first call costs $0.25 and the remaining nineteen cost $0.475 — $0.725 in total, a 6.9x saving — and it survives only if the prefix is byte-identical, which is why injecting a timestamp at the top of a system prompt can quietly multiply a bill by seven.

Token counts are what specification sheets actually specify. "1M context" means 1M tokens, not characters or words: at 1.21 tokens per word that is about 865,000 English words, roughly one and a half times War and Peace. And the ceiling is not always flat — GPT-5.6 prices prompts above 272,000 input tokens at 2x input and 1.5x output, so the last third of the window costs double the first two thirds.

Challenges

The invisible half of the bill. Reasoning tokens are charged at the output rate and never shown. A request that looks like it produced 200 tokens may have been billed for 8,000, and the only way to see it is the usage field in the API response — not the reply. Any cost model built by measuring visible output is wrong for every reasoning model.

Conversation history is a compounding cost. Because the full transcript is resent each turn, a chat that reaches 50 turns of 500 tokens has sent roughly 640,000 input tokens, not 25,000. Long agent loops fail on budget for this reason far more often than they fail on capability.

Counting with the wrong tokenizer. Each model family has its own vocabulary, and a count from one does not transfer to another. Estimating a Claude bill with tiktoken produces a number that is confidently wrong, and the error is largest exactly where it matters: non-English text and code.

Context limits are not word limits. A 128,000-token output cap sounds generous until a task emits JSON, where structural characters, quoted keys and escaped strings drive the token count far above what the equivalent prose would cost. The same content in a compact format can fit where the verbose one truncates.

Truncation is silent. Exceeding an output cap does not raise an error — it returns a response that simply stops, often mid-structure. Code that parses model output without checking the finish reason will read a truncated reply as a complete one.

Code Example

Counting is cheap, estimating is not. Count.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")          # GPT-4o / GPT-5 family

prompt = open("system_prompt.txt").read()
n_in = len(enc.encode(prompt))

# Price the two sides separately — they are not the same number.
IN_PER_M, OUT_PER_M, CACHED_PER_M = 5.00, 30.00, 0.50
expected_out = 800

print(f"input   {n_in:>7} tokens  ${n_in / 1e6 * IN_PER_M:.4f}")
print(f"output  {expected_out:>7} tokens  ${expected_out / 1e6 * OUT_PER_M:.4f}")

# Twenty calls sharing this prefix, with and without caching.
full   = 20 * n_in / 1e6 * IN_PER_M
cached = n_in / 1e6 * IN_PER_M + 19 * n_in / 1e6 * CACHED_PER_M
print(f"20 calls: ${full:.4f} uncached vs ${cached:.4f} cached")

For any model not made by OpenAI, load its own tokenizer instead — AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B") from transformers — and read the usage block of the API response afterwards to see what you were actually billed, including the reasoning tokens no count can predict in advance.