Tokenization

Definition

Tokenization is the step that turns a string into the list of integers a model actually reads. It happens before the model does anything at all, and it is lossy in a way that matters: the unit that survives is neither a word nor a letter, but whatever fragment the tokenizer's fixed vocabulary happens to contain.

Here is the consequence, in one measurement. Under o200k_base, the vocabulary used by OpenAI's recent GPT models, the string " strawberry" (with a leading space) encodes to a single integer: 101830. Not three tokens, not ten characters — one number. When a model fails to count the r's in "strawberry", it is not failing at arithmetic. It is being asked about the spelling of a word it was handed as an opaque symbol.

Measured across 244,000 words of this site's prose, the same encoder averages 5.7 characters per token, or 1.21 tokens per English word. That ratio is the exchange rate between your text and everything downstream: your bill, your latency, and how much fits in the context window. This page is about how that vocabulary gets built and what it costs; Token covers what the resulting units do inside the model.

How It Works

Byte Pair Encoding, the algorithm behind almost every current model, is trained rather than designed. It starts with an alphabet of the 256 possible bytes, counts every adjacent pair of symbols in a training corpus, merges the most frequent pair into a new symbol, and repeats. A vocabulary of 200,019 tokens is 256 bytes plus roughly 199,700 learned merges, in the order they were learned.

Take Sennrich's original worked example: a corpus containing low ×5, lower ×2, newest ×6 and widest ×3. Every word starts as loose characters. The pair (e, s) appears 9 times — in newest and widest — more than any other, so it merges into es. Now (es, t) appears 9 times, and merges into est. Then (l, o) at 7, giving lo, then low. After four merges the vocabulary holds es, est, lo and low, and a word never seen in training — widest — encodes as w + i + d + est. This is the whole trick: frequent things become single tokens, rare things decompose, and nothing is ever unrepresentable.

That last property comes from starting at bytes, not characters. Because the base alphabet is all 256 byte values, any input — an emoji, a Chinese character, a corrupted UTF-8 sequence, a private-use codepoint — has some encoding. Byte-level BPE tokenizers have no [UNK] token, because the case it exists for cannot arise. Older word-level tokenizers did, and every time it fired the model lost information permanently.

Encoding new text applies the learned merges in their learned order; decoding is a lookup table and a byte concatenation, which is why it is essentially free. The interesting decision is not the algorithm but the vocabulary size, and that is a real budget. The embedding table is vocabulary × model dimension parameters. For GPT-2 small, 50,257 × 768 = 38.6M of its 124M parameters — 31% of the entire model spent on a lookup table. For Llama 3 70B, 128,256 × 8,192 = 1.05B against 70.55B, or 1.5%.

That collapse from 31% to 1.5% is why vocabularies quadrupled. When the table was a third of your model, every added token was expensive; once models reached tens of billions of parameters it became rounding error, while each extra token still bought shorter sequences — and sequence length is paid for on every forward pass, with the attention term growing quadratically. Compression stopped costing anything and kept paying.

Types

Three genuine training objectives exist, and they differ in what they consider the best merge.

Byte Pair Encoding (BPE)

Merge by raw frequency, as above. Introduced for translation by Sennrich et al. in 2016 and now the default: GPT-2 onward, Llama, Mistral and most open-weight models use byte-level variants. It is trivial to implement, deterministic, and fast.

WordPiece

Same greedy loop, different scoring: at each step it merges the pair that most increases the likelihood of the training corpus under a unigram language model, which is roughly frequency divided by how common the two halves already are on their own. That penalises merging a common piece into everything it touches. BERT's 30,522-token vocabulary is the canonical example, and the ## prefixes in its output mark continuation pieces.

Unigram

Kudo's 2018 method runs the other direction. It seeds a deliberately oversized candidate vocabulary, then repeatedly deletes the tokens whose removal costs the least corpus likelihood until the target size is reached. Because it keeps a probability per token rather than an ordered merge list, it can score several valid segmentations of the same word — the basis of subword regularization, where sampling different segmentations during training acts as data augmentation. T5, ALBERT and XLNet use it.

What SentencePiece actually is

SentencePiece is frequently listed as a fourth algorithm. It is not — it is a library that implements BPE and Unigram, and you must pick one when you train it. Its real contribution is upstream of both: it consumes text as a raw Unicode stream and never pre-splits on whitespace, encoding spaces as the visible character ▁ instead. That single decision is what makes one tokenizer usable across English, Japanese and Thai, and it makes encoding perfectly reversible — nothing about the original spacing is thrown away before the algorithm sees it.

Real-World Applications

A tokenizer upgrade is a pricing change. Below is the same sentence — "Artificial intelligence is changing how people work." — translated and measured under both of OpenAI's recent encoders. Each column is token count, with its multiple of the English cost in brackets.

A Hindi speaker paid 6.7x an English speaker for the same sentence under GPT-4's vocabulary and got 6.7x less of it into the context window. Doubling the vocabulary cut that to 2.3x. Petrov et al. found gaps as wide as 15x across the language pairs they surveyed, and the effect compounds: it is simultaneously a cost, a latency and a context-length penalty, borne entirely by people who do not write in English.

And the English speaker got almost nothing from it. Running the same 244,000 words of English prose through all three generations of OpenAI encoder gives 5.57 characters per token for GPT-2's 50,257-token vocabulary, 5.65 for cl100k_base at 100,277, and 5.69 for o200k_base at 200,019 — a 2% improvement for a 4x larger vocabulary. Every real gain from that growth went to non-English text and to code. A vocabulary is a budget allocated across the world's writing systems, and English saturated its share three generations ago.

The vocabulary is part of the model's identity, not a detachable preprocessing step. DBRX shipped using GPT-4's tiktoken vocabulary outright rather than training its own. Llama went the other way, abandoning its 32,000-token SentencePiece tokenizer for a 128,256-token tiktoken-style one between versions 2 and 3 — a change that alone improved how much text and code fit per token. In both cases the weights and the tokenizer are married: load a model with the wrong encoder and you do not get degraded output, you get token ids pointing at rows of the embedding table that mean something else entirely.

Challenges

Character-level tasks are structurally impossible. Counting letters, reversing a string, detecting rhyme, spotting a typo — all require access to characters the model was never given. This is why "how many r's in strawberry" became a benchmark: the failure is not a reasoning gap, it is the input representation, and no amount of scale removes it.

Arithmetic inherits the vocabulary's idea of a digit. cl100k_base and o200k_base both chunk numbers three digits at a time, left to right, so 1234 becomes ["123", "4"] and 1000 becomes ["100", "0"]. Place value is destroyed: the 4 in 1234 and the 4 in 4 are the same token despite being different magnitudes, and two numbers of different lengths chunk into misaligned pieces. Singh and Strouse showed that forcing right-to-left chunking by inserting commas — 1,234 tokenizes as ["1", ",", "234"] — improves integer arithmetic accuracy by up to 20% on the same model. The prompt did not get smarter; the digits just lined up.

Glitch tokens are vocabulary entries the model never learned. The tokenizer is trained on a different, usually larger corpus than the model. Strings that were frequent in the tokenizer's corpus but effectively absent from training end up as tokens with near-random embeddings. " SolidGoldMagikarp" — a Reddit username scraped into GPT-2's corpus — is a single token in that vocabulary, and prompting GPT-3 with it produced evasions, insults and non-sequiturs. It decomposes into five ordinary tokens under cl100k_base, which is the fix: the pathology lives in the vocabulary, not the weights.

A trailing space silently moves you off-distribution. "The capital of France is" tokenizes with " is" as its final token — space attached to the following word, as the training data almost always had it. Add one space and you get an extra bare " " token, and the model must now continue with something that does not begin with a space, a pattern it has barely seen. The same effect governs code: four spaces of indentation and a tab produce entirely different token sequences for identical Python.

Domain text is quietly overpriced. A vocabulary fit to web text spends its 200,000 slots on web text, so chemical formulas, ICD codes and protein sequences decompose into many short pieces and cost far more tokens than their length suggests — the case Dagan et al. make for extending the tokenizer during domain adaptation instead of treating it as fixed.

Future Trends

The most interesting direction is deleting the tokenizer. Pagnoni et al.'s Byte Latent Transformer operates on raw bytes grouped into dynamic patches, with boundaries placed where a small byte-level model finds the next byte hard to predict — so patches lengthen over predictable text and shorten where information is dense, instead of being fixed by a vocabulary decided before training began. Every failure above is downstream of that fixed vocabulary, so an architecture without one would resolve them together rather than one at a time.

The narrower, already-shipping trend is treating tokenization as a design decision rather than an inherited default: right-to-left number tokenization in models that care about arithmetic, vocabularies sampled for multilingual balance rather than corpus frequency. Both are cheap at training time and impossible to change afterwards.

Code Example

Every number on this page is reproducible with pip install tiktoken:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
print(enc.n_vocab)  # 200019

# One token, not ten characters.
print(enc.encode(" strawberry"))  # [101830]

# The leading space matters: without it, three tokens.
ids = enc.encode("strawberry")
print([enc.decode([i]) for i in ids])  # ['st', 'raw', 'berry']

# Left-to-right 3-digit chunking breaks place value.
print([enc.decode([i]) for i in enc.encode("1234")])   # ['123', '4']
print([enc.decode([i]) for i in enc.encode("1,234")])  # ['1', ',', '234']

# Measure the exchange rate for your own text.
text = open("some_document.txt").read()
print(len(text) / len(enc.encode(text)), "characters per token")

Swap o200k_base for cl100k_base to reproduce the multilingual table, and use AutoTokenizer.from_pretrained(...) from transformers for non-OpenAI models. Never estimate a token count with a different model's tokenizer.

Academic Sources

• "Neural Machine Translation of Rare Words with Subword Units" — Sennrich et al. (2016), the paper that brought BPE to NLP
• "Google's Neural Machine Translation System" — Wu et al. (2016), WordPiece
• "Subword Regularization" — Kudo (2018), the Unigram method
• "SentencePiece: A simple and language independent subword tokenizer" — Kudo & Richardson (2018)
• "Language Model Tokenizers Introduce Unfairness Between Languages" — Petrov et al. (2023), the multilingual cost gap
• "Tokenization counts: the impact of tokenization on arithmetic" — Singh & Strouse (2024), digit chunking and the comma result
• "Getting the most out of your tokenizer for pre-training and domain adaptation" — Dagan et al. (2024)
• "Byte Latent Transformer: Patches Scale Better Than Tokens" — Pagnoni et al. (2024), dynamic byte patching
• tiktoken — OpenAI's byte-level BPE tokenizer