Chunking

Definition

Chunking is the step in a retrieval pipeline where a long document is cut into smaller passages — typically a few hundred to a thousand tokens each — before every passage is turned into an embedding and stored for search. The size and boundaries of those passages, far more than the cleverness of the model, often decide whether the system later finds the sentence that answers a question or a fragment that is useless.

The reason this matters is a hard constraint that is easy to miss: a retrieval system does not search inside your documents, it searches over the chunks you cut them into. Each chunk becomes exactly one vector, one atomic unit that is either retrieved whole or not at all. Cut the document badly and the right answer may exist in your data yet be unreachable — split across two chunks so that neither one matches the question well, or buried inside a chunk so large that its embedding describes the average of ten topics and stands out for none of them. Chunking is where a retrieval-augmented generation system quietly decides what it will and will not be able to answer, before a single query is ever asked.

How It Works

A document arrives as a long string of text. Before it can be searched semantically, it has to become vectors, and an embedding model turns a passage into a single fixed-length vector — a few hundred to a few thousand numbers that summarize its meaning. You cannot feed a 40-page manual to that model and get back one useful vector: the result would be a blurry average of everything in the manual, close to every question and specific to none. So the document is first split into chunks, each chunk is embedded, and the vectors are stored in an index (usually a vector database) alongside the original text. At query time the question is embedded the same way, the nearest chunk vectors are retrieved, and their text is pasted into the model's prompt.

The central tension is between embedding resolution and context completeness, and it pulls in opposite directions:

• A large chunk carries plenty of surrounding context, so whatever you retrieve is self-contained. But one vector now has to represent many sentences at once. If a 900-token chunk covers installation, licensing, and troubleshooting, its embedding sits at the centroid of all three, and a sharp question about licensing matches it only weakly — the signal is diluted by everything else in the chunk.
• A small chunk produces a crisp, specific embedding that matches a narrow question precisely. But it may no longer contain enough to answer that question. A chunk holding only "This is deprecated in version 3" has lost which feature this refers to, because the noun sat two sentences earlier, in the previous chunk.

There is no size that escapes this tradeoff, which is why any specific "optimal chunk size" you read is a claim about one dataset and one embedding model, not a law. It is tuned, not derived.

Overlap

Overlap is the standard defense against the boundary problem. Instead of cutting cleanly at token 512 and starting the next chunk at 513, you start the next chunk a little earlier — repeating, say, the last 15% of the previous chunk. A fact that would have been severed by the boundary now appears whole in at least one chunk. The cost is duplication: you store and embed more tokens than the document actually contains, and the same passage can be retrieved twice.

A worked example

Take a 40-page document. Estimating text length in tokens uses a common rule of thumb — roughly four characters, or about 0.75 words, per token for English — so at about 500 words per page:

40 pages x 500 words = 20,000 words 20,000 words / 0.75 words-per-token = ~26,666 tokens

Now chunk it at 512 tokens with 15% overlap. Fifteen percent of 512 is about 76 tokens, so each new chunk advances by a stride of 512 − 76 = 436 tokens rather than a full 512:

number of chunks = ceil(26,666 / 436) = 62 chunks

With no overlap the same document needs only ceil(26,666 / 512) = 53 chunks. The 15% overlap did not add 15% more chunks; it added about 17%, because what grows the count is the shrunken stride (512 → 436), a ratio of 1.18. You also embed more tokens than you have: those 62 chunks contain 31,296 tokens against the document's 26,666 — a 17.4% inflation, the price of every fact surviving a boundary.

Embedding is a one-time cost, paid once per document unless you re-chunk. It is also cheap per document but linear in corpus size: at a representative commodity embedding rate on the order of $0.02 per million tokens (roughly the going rate in mid-2026, and the part of this example most likely to age), those ~31,000 tokens cost about six-hundredths of a cent. The number only matters at scale — 100,000 documents of this size is about 3.1 billion tokens, on the order of $60 to embed the corpus once. Chunk size and overlap set that bill, and re-chunking a large corpus means paying it again.

Types

Chunking strategies form a genuine progression, from ignoring the text's structure to reading it. These are named, widely-implemented approaches, not a taxonomy invented for this page.

Fixed-size chunking cuts every N tokens (or characters), usually with overlap, and ignores what the text says. It is the simplest and most predictable method, cheap to run and easy to reason about, and it is the baseline most pipelines start with. Its weakness is that a boundary can land in the middle of a sentence, a table, or a code block, because the splitter counts tokens and nothing else.

Recursive chunking respects the document's natural separators in priority order. It tries to split on the largest boundary first — paragraphs — and only if a piece is still too big does it fall back to sentences, then words, then raw characters. LangChain's widely-used RecursiveCharacterTextSplitter is the canonical implementation. The result keeps most chunks aligned to paragraph and sentence breaks while still capping their size, which is why it is the common default for general prose.

Semantic chunking places boundaries where the meaning shifts rather than where a token count runs out. A typical implementation embeds each sentence, walks through the document comparing consecutive sentences, and starts a new chunk wherever the similarity between neighbors drops sharply — the signal that the topic has changed. Each chunk then holds one coherent idea. The catch is cost and circularity: you must embed the document to decide how to chunk it, and the quality depends on the sentence splitter and the similarity threshold.

Structure-aware (layout) chunking uses the document's own markup — Markdown headings, HTML tags, PDF layout, or a code file's function boundaries — as the split points. Chunking a codebase by function or class, or a manual by its section headings, keeps each chunk to a unit a human already treats as whole. Libraries such as Unstructured specialize in extracting this structure from messy PDFs and Office files so it can be chunked on.

Real-World Applications

Every production RAG system makes a chunking decision, and the tooling makes it explicit. LangChain and LlamaIndex, the two most common orchestration frameworks, both ship a family of splitters — fixed-size, recursive, sentence-window, and semantic — as a first-class configuration step, because teams found chunking to be one of the highest-leverage knobs in the pipeline. Unstructured exists largely to turn PDFs, slide decks, and scanned documents into clean, layout-aware chunks before embedding, which is the unglamorous majority of real-world ingestion work.

The decision changes shape by domain. A documentation or customer-support assistant over a knowledge base typically chunks by section heading so each answer is grounded in one self-contained topic. A code assistant searching a repository chunks by function or class rather than by token count, because half a function is rarely a useful retrieval unit. A legal or financial RAG system over contracts has to keep a clause and its defined terms together, which pushes toward structure-aware splitting and larger overlap, since a severed exception clause is not a minor error there.

Chunking also constrains what comes after retrieval. Because each retrieved chunk consumes prompt tokens, systems retrieve only a handful (a small top-k), and a downstream reranking step — a second, more precise model that reorders the retrieved chunks — can only work with the chunks it is handed. If the right passage was cut badly and never retrieved, no reranker recovers it. Chunking sets the ceiling on everything downstream.

Key Concepts

• Chunk size: the target length of each passage, in tokens or characters. It is the primary lever on the resolution-versus-context tradeoff, and it is dataset-specific, not universal.
• Overlap and stride: overlap is the text shared between adjacent chunks; stride is how far the window advances (chunk size minus overlap). Stride, not overlap directly, determines how many chunks you get.
• The one-vector-per-chunk constraint: an embedding model compresses a whole chunk into a single point. Everything the chunk contains is averaged into that point, which is why a chunk holding several topics retrieves poorly for any one of them.
• Top-k coupling: retrieval fetches the k nearest chunks, and each one costs context-window tokens in the prompt. Bigger chunks with the same k means more tokens per query and higher latency and cost on every call.

Challenges

The failures are specific to chunking, and most are invisible until a query exposes them.

Boundaries sever facts. The defining failure mode: because the split happens before anyone knows what will be asked, a fact spanning a boundary — a figure in one chunk and its caption in the next, a subject and the pronoun that refers to it — ends up in neither chunk whole. Overlap reduces this but cannot eliminate it, because a fact longer than the overlap still gets cut.

There is no optimal size to find. Tuning chunk size is not a search for a hidden right answer; the best size depends on the embedding model, the document type, and the questions users actually ask, and it changes when any of those change. A size tuned on tidy documentation will underperform on dense contracts in the same index.

Uniform rules break on non-prose. Fixed-size and even recursive splitters assume flowing text. Tables get cut mid-row, code gets cut mid-function, and a list gets separated from the sentence introducing it. These structures need their own handling, which is why layout-aware parsing exists.

Re-chunking is expensive and easy to forget. Chunk size is baked into the stored vectors. Changing it means re-embedding the entire corpus and rebuilding the index — the ~$60-per-100k-documents cost from the worked example, paid again — so a chunking choice made early is one you live with far longer than expected.

Chunking quality is hard to measure directly. A bad chunking strategy does not raise an error; it lowers answer quality in ways that only show up in end-to-end retrieval evaluation. Teams often discover a chunking problem only by auditing why specific questions returned the wrong passage.

Future Trends

Late chunking inverts the usual order. Instead of splitting the document and then embedding each piece, it embeds the whole document with a long-context model first — so every token's representation is informed by the full document — and only then pools those token representations into chunk vectors. Boundaries no longer cut off the surrounding context, because the context was already baked into each token before the split. It trades a heavier embedding step for chunks that no longer suffer the severed-fact problem as sharply.

LLM-assisted and agentic chunking use a language model to decide the boundaries — asking a model to segment a document into coherent, self-contained passages, or to write a one-line summary that travels with each chunk. This produces cleaner splits than a similarity threshold but costs a model call per document, so its viability tracks how cheap inference becomes.

Longer context windows change the balance without removing the need. As models accept larger prompts, systems can retrieve more and larger chunks, and some workloads shrink toward "retrieve a few big sections." But embedding still compresses each chunk into one vector, so search resolution still argues for splitting, and cost and latency still argue against stuffing everything in. Longer context moves the tradeoff; it does not abolish it.

Code Example

Fixed-size chunking with overlap is a few lines, and writing it out makes the stride arithmetic concrete. This splitter operates on a list of tokens (a real pipeline would use the embedding model's own tokenizer):

def chunk(tokens, size=512, overlap_frac=0.15):
    overlap = int(size * overlap_frac)   # 76 tokens
    stride  = size - overlap             # 436 tokens
    out, start = [], 0
    while start < len(tokens):
        out.append(tokens[start:start + size])
        start += stride
    return out

# 40 pages x ~500 words/page x ~1.33 tokens/word (the ~0.75 words-per-token rule)
n_tokens = 40 * 500 * 4 // 3             # 26,666 tokens
doc = list(range(n_tokens))

overlapped = chunk(doc, 512, 0.15)
no_overlap = chunk(doc, 512, 0.0)

print("raw tokens:", n_tokens)
print("chunks with 15% overlap:", len(overlapped))
print("chunks with no overlap :", len(no_overlap))
print("tokens embedded (overlap):", sum(len(c) for c in overlapped))

Running this prints:

raw tokens: 26666
chunks with 15% overlap: 62
chunks with no overlap : 53
tokens embedded (overlap): 31296

The whole strategy lives in one line — stride = size - overlap. Drop the overlap to zero and you get 53 tidy, non-repeating chunks that will occasionally cut a fact in half. Add it back and you get 62 chunks and 31,296 embedded tokens against a 26,666-token document, buying boundary safety with a 17% surcharge you pay on storage, on re-embedding, and on every duplicate a query pulls back.