---
source: 'https://howaiworks.ai/glossary/tokenization'
section: glossary
title: Tokenization
description: >-
  How text becomes the integers a language model reads: BPE, WordPiece and
  Unigram, why " strawberry" is one token, and what that breaks.
tags:
  - NLP
  - LLM
  - Multilingual AI
  - Tokens
category: Natural Language Processing
datePublished: '2025-07-16'
lastUpdated: '2026-07-21'
---

# Tokenization

> How text becomes the integers a language model reads: BPE, WordPiece and Unigram, why " strawberry" is one token, and what that breaks.

## Definition

Tokenization is the step that turns a string into the list of integers a model actually reads. It
happens before the model does anything at all, and it is lossy in a way that matters: the unit that
survives is neither a word nor a letter, but whatever fragment the tokenizer's fixed vocabulary
happens to contain.

Here is the consequence, in one measurement. Under `o200k_base`, the vocabulary used by OpenAI's
recent [GPT](https://howaiworks.ai/models/gpt) models, the string `" strawberry"` (with a leading space) encodes to a
single integer: **101830**. Not three tokens, not ten characters — one number. When a model fails to
count the r's in "strawberry", it is not failing at arithmetic. It is being asked about the spelling
of a word it was handed as an opaque symbol.

Measured across 244,000 words of this site's prose, the same encoder averages **5.7 characters per
token, or 1.21 tokens per English word**. That ratio is the exchange rate between your text and
everything downstream: your bill, your latency, and how much fits in the
[context window](https://howaiworks.ai/glossary/context-window). This page is about how that vocabulary gets built and
what it costs; [Token](https://howaiworks.ai/glossary/token) covers what the resulting units do inside the model.

## How It Works

Byte Pair Encoding, the algorithm behind almost every current model, is trained rather than
designed. It starts with an alphabet of the 256 possible bytes, counts every adjacent pair of
symbols in a training corpus, merges the most frequent pair into a new symbol, and repeats. A
vocabulary of 200,019 tokens is 256 bytes plus roughly 199,700 learned merges, in the order they
were learned.

Take Sennrich's original worked example: a corpus containing `low` ×5, `lower` ×2, `newest` ×6 and
`widest` ×3. Every word starts as loose characters. The pair `(e, s)` appears 9 times — in `newest`
and `widest` — more than any other, so it merges into `es`. Now `(es, t)` appears 9 times, and
merges into `est`. Then `(l, o)` at 7, giving `lo`, then `low`. After four merges the vocabulary
holds `es`, `est`, `lo` and `low`, and a word never seen in training — `widest` — encodes as
`w` + `i` + `d` + `est`. This is the whole trick: frequent things become single tokens, rare things
decompose, and nothing is ever unrepresentable.

**That last property comes from starting at bytes, not characters.** Because the base alphabet is
all 256 byte values, any input — an emoji, a Chinese character, a corrupted UTF-8 sequence, a
private-use codepoint — has *some* encoding. Byte-level BPE tokenizers have no `[UNK]` token,
because the case it exists for cannot arise. Older word-level tokenizers did, and every time it
fired the model lost information permanently.

Encoding new text applies the learned merges in their learned order; decoding is a lookup table and
a byte concatenation, which is why it is essentially free. The interesting decision is not the
algorithm but the vocabulary size, and that is a real budget. The embedding table is
`vocabulary × model dimension` [parameters](https://howaiworks.ai/glossary/parameters). For GPT-2 small, 50,257 × 768 =
**38.6M of its 124M parameters — 31% of the entire model** spent on a lookup table. For
[Llama 3](https://howaiworks.ai/models/llama) 70B, 128,256 × 8,192 = 1.05B against 70.55B, or **1.5%**.

That collapse from 31% to 1.5% is why vocabularies quadrupled. When the table was a third of your
model, every added token was expensive; once models reached tens of billions of parameters it became
rounding error, while each extra token still bought shorter sequences — and sequence length is paid
for on every forward pass, with the [attention](https://howaiworks.ai/glossary/transformer) term growing quadratically.
Compression stopped costing anything and kept paying.

## Types

Three genuine training objectives exist, and they differ in what they consider the best merge.

### Byte Pair Encoding (BPE)

Merge by raw frequency, as above. Introduced for translation by Sennrich et al. in 2016 and now the
default: GPT-2 onward, Llama, Mistral and most open-weight models use byte-level variants. It is
trivial to implement, deterministic, and fast.

### WordPiece

Same greedy loop, different scoring: at each step it merges the pair that most increases the
likelihood of the training corpus under a unigram language model, which is roughly frequency divided
by how common the two halves already are on their own. That penalises merging a common piece into
everything it touches. BERT's 30,522-token vocabulary is the canonical example, and the `##`
prefixes in its output mark continuation pieces.

### Unigram

Kudo's 2018 method runs the other direction. It seeds a deliberately oversized candidate vocabulary,
then repeatedly deletes the tokens whose removal costs the least corpus likelihood until the target
size is reached. Because it keeps a probability per token rather than an ordered merge list, it can
score *several* valid segmentations of the same word — the basis of subword regularization, where
sampling different segmentations during training acts as data augmentation. T5, ALBERT and XLNet use
it.

### What SentencePiece actually is

SentencePiece is frequently listed as a fourth algorithm. It is not — it is a library that
implements BPE and Unigram, and you must pick one when you train it. Its real contribution is
upstream of both: it consumes text as a raw Unicode stream and never pre-splits on whitespace,
encoding spaces as the visible character `▁` instead. That single decision is what makes one
tokenizer usable across English, Japanese and Thai, and it makes encoding perfectly reversible —
nothing about the original spacing is thrown away before the algorithm sees it.

## Real-World Applications

**A tokenizer upgrade is a pricing change.** Below is the same sentence — *"Artificial intelligence
is changing how people work."* — translated and measured under both of OpenAI's recent encoders.
Each column is token count, with its multiple of the English cost in brackets.

| Language | `cl100k_base` (GPT-4) | `o200k_base` |
|---|---|---|
| English | 9 (1.0x) | 8 (1.0x) |
| Spanish | 16 (1.8x) | 14 (1.8x) |
| Russian | 25 (2.8x) | 13 (1.6x) |
| Japanese | 22 (2.4x) | 17 (2.1x) |
| Hindi | 60 (6.7x) | 18 (2.3x) |
| Burmese | 104 (11.6x) | 28 (3.5x) |

A Hindi speaker paid **6.7x** an English speaker for the same sentence under GPT-4's vocabulary and
got 6.7x less of it into the context window. Doubling the vocabulary cut that to 2.3x. Petrov et
al. found gaps as wide as **15x** across the language pairs they surveyed, and the effect compounds:
it is simultaneously a cost, a latency and a context-length penalty, borne entirely by people who
do not write in English.

**And the English speaker got almost nothing from it.** Running the same 244,000 words of English
prose through all three generations of OpenAI encoder gives 5.57 characters per token for GPT-2's
50,257-token vocabulary, 5.65 for `cl100k_base` at 100,277, and 5.69 for `o200k_base` at 200,019 —
**a 2% improvement for a 4x larger vocabulary**. Every real gain from that growth went to
non-English text and to code. A vocabulary is a budget allocated across the world's writing systems,
and English saturated its share three generations ago.

**The vocabulary is part of the model's identity, not a detachable preprocessing step.**
[DBRX](https://howaiworks.ai/models/dbrx) shipped using GPT-4's tiktoken vocabulary outright rather than training its
own. Llama went the other way, abandoning its 32,000-token SentencePiece tokenizer for a
128,256-token tiktoken-style one between versions 2 and 3 — a change that alone improved how much
text and code fit per token. In both cases the weights and the tokenizer are married: load a model
with the wrong encoder and you do not get degraded output, you get token ids pointing at rows of the
embedding table that mean something else entirely.

## Challenges

**Character-level tasks are structurally impossible.** Counting letters, reversing a string,
detecting rhyme, spotting a typo — all require access to characters the model was never given. This
is why "how many r's in strawberry" became a benchmark: the failure is not a reasoning gap, it is
the input representation, and no amount of scale removes it.

**Arithmetic inherits the vocabulary's idea of a digit.** `cl100k_base` and `o200k_base` both chunk
numbers three digits at a time, left to right, so `1234` becomes `["123", "4"]` and `1000` becomes
`["100", "0"]`. Place value is destroyed: the `4` in `1234` and the `4` in `4` are the same token
despite being different magnitudes, and two numbers of different lengths chunk into misaligned
pieces. Singh and Strouse showed that forcing right-to-left chunking by inserting commas — `1,234`
tokenizes as `["1", ",", "234"]` — improves integer arithmetic accuracy by **up to 20%** on the same
model. The prompt did not get smarter; the digits just lined up.

**Glitch tokens are vocabulary entries the model never learned.** The tokenizer is trained on a
different, usually larger corpus than the model. Strings that were frequent in the tokenizer's
corpus but effectively absent from training end up as tokens with near-random embeddings.
`" SolidGoldMagikarp"` — a Reddit username scraped into GPT-2's corpus — is a **single token** in
that vocabulary, and prompting GPT-3 with it produced evasions, insults and non-sequiturs. It
decomposes into five ordinary tokens under `cl100k_base`, which is the fix: the pathology lives in
the vocabulary, not the weights.

**A trailing space silently moves you off-distribution.** `"The capital of France is"` tokenizes
with `" is"` as its final token — space attached to the following word, as the training data almost
always had it. Add one space and you get an extra bare `" "` token, and the model must now continue
with something that does *not* begin with a space, a pattern it has barely seen. The same effect
governs code: four spaces of indentation and a tab produce entirely different token sequences for
identical Python.

**Domain text is quietly overpriced.** A vocabulary fit to web text spends its 200,000 slots on web
text, so chemical formulas, ICD codes and protein sequences decompose into many short pieces and
cost far more tokens than their length suggests — the case Dagan et al. make for extending the
tokenizer during domain adaptation instead of treating it as fixed.

## Future Trends

The most interesting direction is deleting the tokenizer. Pagnoni et al.'s Byte Latent Transformer
operates on raw bytes grouped into *dynamic* patches, with boundaries placed where a small byte-level
model finds the next byte hard to predict — so patches lengthen over predictable text and shorten
where information is dense, instead of being fixed by a vocabulary decided before training began.
Every failure above is downstream of that fixed vocabulary, so an architecture without one would
resolve them together rather than one at a time.

The narrower, already-shipping trend is treating tokenization as a design decision rather than an
inherited default: right-to-left number tokenization in models that care about arithmetic,
vocabularies sampled for multilingual balance rather than corpus frequency. Both are cheap at
training time and impossible to change afterwards.

## Code Example

Every number on this page is reproducible with `pip install tiktoken`:

```python
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
print(enc.n_vocab)  # 200019

# One token, not ten characters.
print(enc.encode(" strawberry"))  # [101830]

# The leading space matters: without it, three tokens.
ids = enc.encode("strawberry")
print([enc.decode([i]) for i in ids])  # ['st', 'raw', 'berry']

# Left-to-right 3-digit chunking breaks place value.
print([enc.decode([i]) for i in enc.encode("1234")])   # ['123', '4']
print([enc.decode([i]) for i in enc.encode("1,234")])  # ['1', ',', '234']

# Measure the exchange rate for your own text.
text = open("some_document.txt").read()
print(len(text) / len(enc.encode(text)), "characters per token")
```

Swap `o200k_base` for `cl100k_base` to reproduce the multilingual table, and use
`AutoTokenizer.from_pretrained(...)` from `transformers` for non-OpenAI models. Never estimate a
token count with a different model's tokenizer.

## Academic Sources

- ["Neural Machine Translation of Rare Words with Subword Units"](https://arxiv.org/abs/1508.07909) — Sennrich et al. (2016), the paper that brought BPE to NLP
- ["Google's Neural Machine Translation System"](https://arxiv.org/abs/1609.08144) — Wu et al. (2016), WordPiece
- ["Subword Regularization"](https://arxiv.org/abs/1804.10959) — Kudo (2018), the Unigram method
- ["SentencePiece: A simple and language independent subword tokenizer"](https://arxiv.org/abs/1808.06226) — Kudo & Richardson (2018)
- ["Language Model Tokenizers Introduce Unfairness Between Languages"](https://arxiv.org/abs/2305.15425) — Petrov et al. (2023), the multilingual cost gap
- ["Tokenization counts: the impact of tokenization on arithmetic"](https://arxiv.org/abs/2402.14903) — Singh & Strouse (2024), digit chunking and the comma result
- ["Getting the most out of your tokenizer for pre-training and domain adaptation"](https://arxiv.org/abs/2402.01035) — Dagan et al. (2024)
- ["Byte Latent Transformer: Patches Scale Better Than Tokens"](https://arxiv.org/abs/2412.09871) — Pagnoni et al. (2024), dynamic byte patching
- [tiktoken](https://github.com/openai/tiktoken) — OpenAI's byte-level BPE tokenizer

## Frequently Asked Questions

### How many tokens is my text?

For ordinary English prose, about 5.7 characters per token — roughly 1,200 tokens per 1,000 words, measured across 244,000 words with o200k_base. Code, rare names and non-English text run shorter per token, so the only reliable answer is to run your actual string through your model's tokenizer.

### Why can't a language model count the letters in "strawberry"?

Because it never sees the letters. In OpenAI's o200k_base vocabulary the string " strawberry" is a single token, id 101830. The model receives one integer, not ten characters, so counting the r's is a question about a spelling it was never shown.

### What is the difference between BPE, WordPiece and Unigram?

BPE repeatedly merges the most frequent adjacent pair. WordPiece merges the pair that most increases the training corpus's likelihood, not the most common one. Unigram works backwards — it starts from a large candidate vocabulary and prunes the tokens whose removal costs the least likelihood.

### Is SentencePiece a tokenization algorithm?

No. SentencePiece is a library that implements BPE and Unigram. Its real contribution is treating input as a raw Unicode stream with no splitting on spaces, which is what lets the same tokenizer work on Japanese and Thai as on English.

### Why does the same sentence cost more in some languages?

Because the vocabulary was fit to a corpus that was mostly English. One test sentence measured 8 tokens in English and 28 in Burmese under o200k_base — and 104 under the older cl100k_base. Same meaning, 3.5x the bill and 3.5x the context window consumed.

### How large is a modern vocabulary?

GPT-2 used 50,257 tokens, GPT-4's cl100k_base 100,277, and o200k_base 200,019. Llama went from 32,000 in version 2 to 128,256 in version 3. Vocabularies grew because a bigger embedding table became cheap relative to the rest of the model.

## Related

### Related terms

- [Token](https://howaiworks.ai/glossary/token)
- [Context Window](https://howaiworks.ai/glossary/context-window)
- [Embedding](https://howaiworks.ai/glossary/embedding)
- [Large Language Model (LLM)](https://howaiworks.ai/glossary/large-language-model)
- [Transformer](https://howaiworks.ai/glossary/transformer)
- [Natural Language Processing (NLP)](https://howaiworks.ai/glossary/natural-language-processing)

---

Source: https://howaiworks.ai/glossary/tokenization — HowAIWorks.ai