Token

Definition

A token is one entry from a model's fixed vocabulary — most often a fragment of a word — and it is the only thing an AI model ever sees. Your text is converted into a list of integers before the model runs, the model does arithmetic on those integers, and integers come back out. The words you read were never in the building.

That conversion is not a detail, because everything is counted in tokens: what you pay, what fits in the context window, how much the model may write back, and how much GPU memory your session occupies while you hold it. Measured across the 437,000 words of prose in this site's glossary with OpenAI's o200k_base encoder, the exchange rate is 4.65 characters per token, or 1.29 tokens per English word — so 1,000 words costs roughly 1,290 tokens. The old rule of thumb that a token is four characters, which comes from GPT-3's tokenizer, survives better than it deserves to: it understates a modern encoder by about 15%, and the companion rule that a token is three-quarters of a word lands within 3% of the measurement. Rerun it on your own text before trusting either, because the average is corpus-shaped — the same encoder returns 1.35 tokens per word on this site's blog and 1.07 on plain unformatted English.

This page is about the units and what they cost. Tokenization is about how the vocabulary that produces them gets built.

How It Works

A token id is an index. The model holds an embedding table with one row per vocabulary entry, so token 101830 means "fetch row 101830" — a vector of a few thousand numbers that is the model's learned representation of that fragment. Every layer transforms that vector in the context of its neighbours, and the final layer produces a score for every entry in the vocabulary. Sampling one of them yields the next token, which is appended to the sequence, and the whole thing runs again.

That loop is the reason output costs more than input. A prompt of any length is processed in a single pass, because all its tokens are known at once and the arithmetic parallelises across them. Output has no such luxury: token 400 cannot be computed until token 399 exists, so a 500-token reply means 500 sequential passes through every layer of the model. The pricing reflects the physics — GPT-6 Astra charges $10 per million input tokens against $50 per million output, Claude Opus 5 $5 against $25, and Gemini 3.8 Flash $0.75 against $3.75 on its introductory rate. Three vendors, a 13x spread in absolute price, and the identical 5:1 ratio in all three, because they are all paying for the same asymmetry. The outliers are informative rather than contradictory: Grok 4.6 at $2 against $6 for prompts under 200,000 tokens is 3:1, which tells you it is priced to win high-output agent work.

Tokens also occupy memory for as long as you keep them. Each token already processed leaves behind cached key and value vectors so it does not have to be recomputed — the KV cache. For Llama 3.1 70B that costs 320 KiB per token, so a single user at a 128K context is holding 42.9 GB of GPU memory, more than half an H100. A token is not a character with a price tag; it is a character with a price tag and a rent.

Types

The categories below are not a conceptual scheme — they are the line items on an API invoice, and mistaking one for another is how estimates go wrong by an order of magnitude.

Input tokens

Everything you send: system prompt, conversation history, retrieved documents, tool definitions and tool results. In a multi-turn conversation the entire history is resent on every request, so input grows quadratically with turn count even though each individual message is small.

Output tokens

Everything the model generates, billed at 3-6x the input rate and most often at exactly 5x. Providers cap them separately from the context window, and often cap input separately too: GPT-6 Astra has a 1,050,000-token context of which at most 922,000 may be input, and it will emit at most 128,000 whatever room is left. The three numbers are three different limits and only one of them is the headline.

Reasoning tokens

Tokens a reasoning model produces for itself before answering. They are billed as output, they consume the output cap — on GPT-6 Astra the same 128,000 — and they are not returned to you. A request can therefore fail to produce a visible answer while having been charged in full, which is a failure mode that did not exist before reasoning models.

Cached tokens

Input the provider has already processed and can replay cheaply. GPT-6 Astra bills cached input at $1.00 per million against $10.00 — a 10x discount — provided the prefix is byte-identical. It also charges $12.50 per million to write the cache, which is 1.25x the uncached rate, so the discount is a loan you repay on the first call and collect on every one after. Gemini 3.8 Flash prices caching as rent rather than as a discount: $0.075 per million tokens read, plus $0.50 per million tokens per hour of storage. Anthropic sits between the two, with cache reads at 0.1x the base input rate and a 5-minute write at 1.25x.

Special and control tokens

Vocabulary entries that mean something structural rather than lexical. BERT uses [CLS], [SEP], [MASK] and [PAD]; chat models use control tokens such as <|im_start|> to mark where the system prompt ends and the user's turn begins. This is why a chat request is not simply your strings concatenated: the template inserts control tokens, and they are billed like any other.

Tokens that are not text

Images, audio and video are converted to tokens too, and billed as input. The count tracks resolution — Claude Sonnet 5 raised its limit from 1568 to 2576 pixels on the long edge, and a high-resolution image can consume roughly 3x more image tokens as a result. A screenshot pasted into a prompt is not free context.

Real-World Applications

Which side of the invoice dominates depends entirely on the workload, and it flips. Take GPT-6 Astra at $10 in and $50 out per million, and two jobs of similar size:

The first row is 50,000 × $10 ÷ 1,000,000 = $0.500 of input against 500 × $50 ÷ 1,000,000 = $0.025 of output: a $0.525 bill that is 95% prompt. The second is $0.020 against $0.200 — a $0.220 bill that is 91% reply. Same model, same price list, opposite conclusion about what to optimise. Shortening prompts is wasted effort on the second job; shortening replies is wasted effort on the first. You cannot know which you are running without counting both sides.

Caching changes the arithmetic more than model choice does. Ask twenty questions about the same 50,000-token document and you send a million input tokens: 1,000,000 × $10 ÷ 1,000,000 = $10.00 at Astra's full rate. Cache the prefix instead and the first call pays the write rate, 50,000 × $12.50 ÷ 1,000,000 = $0.625, while the remaining nineteen read it at 950,000 × $1.00 ÷ 1,000,000 = $0.950. Total $1.575 — a 6.3x saving on an identical workload.

The write rate is the part people miss. At $12.50 per million it is 1.25x the uncached price, so caching a prefix you use exactly once costs $0.625 where sending it plainly costs $0.500: you have paid 25% extra for a discount you never collect. Caching pays from the second call onward, and only while the prefix stays byte-identical — which is why injecting a timestamp at the top of a system prompt can quietly multiply a bill by six.

Token counts are what specification sheets actually specify. "1M context" means 1M tokens, not characters or words: at 1.29 tokens per word that is about 775,000 English words, seven or eight full-length novels. Read the sub-limits too. Astra's 1,050,000-token context admits at most 922,000 input tokens — about 715,000 words — and reserves the rest for a reply capped at 128,000.

And the ceiling is not always flat. Astra prices any request whose input exceeds 272,000 tokens at 2x the input and cache rates and 1.5x the output rate — for the entire request, not just the tokens above the line. That distinction is the whole lesson. A 271,000-token prompt bills at 271,000 × $10 ÷ 1,000,000 = $2.71 of input. Add two thousand tokens and the 273,000-token prompt reprices end to end at $20 per million: 273,000 × $20 ÷ 1,000,000 = $5.46. Those 2,000 extra tokens — 0.7% more input — cost $2.75 between them, an effective $1,375 per million, or 137.5x the sticker rate. Output crosses at the same instant, from $50 to $75 per million, and cache reads from $1.00 to $2.00.

The practical consequence is that a prompt hovering near the line is worth trimming rather than tolerating, and a retrieval step that sometimes returns 280,000 tokens has a bimodal bill nobody budgeted for. The shape is not unique to OpenAI: Grok 4.6 reprices the whole request at or above 200,000 tokens, and GPT-5.6 carries Astra's rule at the same 272,000-token threshold.

Challenges

The invisible half of the bill. Reasoning tokens are charged at the output rate and never shown. A request that looks like it produced 200 tokens may have been billed for 8,000, and the only way to see it is the usage field in the API response — not the reply. Any cost model built by measuring visible output is wrong for every reasoning model.

Conversation history is a compounding cost. Because the full transcript is resent each turn, a chat that reaches 50 turns of 500 tokens has sent roughly 640,000 input tokens, not 25,000. Long agent loops fail on budget for this reason far more often than they fail on capability.

The rate card is not a constant, and it moves both ways. When Astra launched on 4 September 2026 OpenAI cut the previous generation's prices underneath it: GPT-5.6 Sol went from $5/$30 per million to $4/$20 — 20% off input and 33% off output — and GPT-5.6 Luna from $1/$6 to $0.20/$1.20, an 80% cut on both sides. A unit-economics model written in August overstated a September bill by a fifth to a third, and the cut left GPT-5.5, untouched at $5/$30, costing more than the newer model that supersedes it. Prices move up too: Gemini 3.8 Flash's $0.75/$3.75 is an introductory rate that doubles to $1.50/$7.50 on 1 January 2027. Treat any per-token figure, including the ones on this page, as a reading rather than a constant.

Counting with the wrong tokenizer. Each model family has its own vocabulary, and a count from one does not transfer to another. Estimating a Claude bill with tiktoken produces a number that is confidently wrong, and the error is largest exactly where it matters: non-English text and code.

Context limits are not word limits. A 128,000-token output cap sounds generous until a task emits JSON, where structural characters, quoted keys and escaped strings drive the token count far above what the equivalent prose would cost. The same content in a compact format can fit where the verbose one truncates.

Truncation is silent. Exceeding an output cap does not raise an error — it returns a response that simply stops, often mid-structure. Code that parses model output without checking the finish reason will read a truncated reply as a complete one.

Code Example

Counting is cheap, estimating is not. Count — and price all four rates, not two.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")     # GPT-4o / GPT-5 / GPT-6 family

prompt = open("system_prompt.txt").read()
n_in = len(enc.encode(prompt))

# GPT-6 Astra, dollars per million tokens.
IN, OUT, CACHE_READ, CACHE_WRITE = 10.00, 50.00, 1.00, 12.50
CLIFF = 272_000        # above this the WHOLE request reprices, not just the overage
expected_out = 800

over = n_in > CLIFF
m_in = 2.0 if over else 1.0        # input and cache rates double
m_out = 1.5 if over else 1.0       # output goes to 1.5x

print(f"input   {n_in:>7} tokens  ${n_in / 1e6 * IN * m_in:.4f}")
print(f"output  {expected_out:>7} tokens  ${expected_out / 1e6 * OUT * m_out:.4f}")

# Twenty calls sharing this prefix: uncached, versus one cache write and nineteen reads.
uncached = 20 * n_in / 1e6 * IN * m_in
cached = n_in / 1e6 * CACHE_WRITE * m_in + 19 * n_in / 1e6 * CACHE_READ * m_in
print(f"20 calls: ${uncached:.4f} uncached vs ${cached:.4f} cached")

On a 50,000-token prompt that prints $0.5000 of input, $0.0400 of output, and 20 calls: $10.0000 uncached vs $1.5750 cached — the same arithmetic worked above. Push n_in past 272,000 and every one of those figures changes, which is the point of keeping the threshold in the code rather than in your head.

For any model not made by OpenAI, load its own tokenizer instead — AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B") from transformers — and read the usage block of the API response afterwards to see what you were actually billed, including the reasoning tokens no count can predict in advance.

---

Note: the prices on this page were last verified on 6 September 2026 against OpenAI's GPT-6 Astra model reference, Anthropic's API pricing page and Google's Gemini pricing. Per-token rates, context limits and surcharge thresholds are the parts of this page most likely to age; the mechanisms they illustrate are not.