Chunking

Lesson 3 of 4 in RAG Mechanics: Embeddings and Search.

Nobody embeds whole documents, and the reasons are structural, not stylistic. First, one text becomes one vector — that is the bi-encoder deal. Squeeze a 40-page policy manual into a single point in embedding space and you get the average of everything it discusses; a query about one specific clause has nothing sharp to land on. Second, embedding models accept a bounded input length — long documents must be split before the model will even read them. Third, the chunk is what retrieval ultimately delivers: it is the unit that lands in the Context window. You want pieces small enough that several independent bits of evidence fit the budget, and large enough that each piece is readable evidence rather than an orphaned sentence.

So Chunking is not preprocessing trivia — it fixes the granularity of meaning your whole retrieval system can resolve. Every choice downstream, from index size to answer quality, inherits it.

Fixed-size + overlap

Split every N tokens (a few hundred is a common starting point), with consecutive chunks overlapping by roughly 10–20%. Dumb, predictable, and works on anything — which is precisely its value when the corpus has no reliable structure: transcripts, chat logs, OCR output.

The overlap is not decoration. A hard cut every N tokens will eventually land mid-sentence, mid-table, mid-answer; overlap ensures the severed thought appears intact in at least one chunk, at the price of some duplicated storage. Count in tokens, not characters — the embedding model’s input limit is denominated in its own tokenizer’s tokens.

Structure-aware

Split where the author already did: headings, paragraphs, list items, numbered clauses, code blocks. Documents with real structure — Markdown wikis, API references, contracts — come pre-segmented into units that match how people ask questions, and cutting along those seams keeps each chunk about one thing.

Chunk sizes become uneven, so very long sections usually get a fixed-size fallback inside them. Bonus: the structure itself (the heading path, the clause number) is high-value metadata — keep it.

Semantic

Let the embeddings draw the boundaries: embed sentences or sliding windows at ingest, and split where similarity between adjacent stretches drops — a measurable topic shift. Useful when structure is absent and fixed windows keep cutting arguments in half: long-form essays, meandering reports.

The cost is an extra embedding pass and a threshold to tune at ingest time. Treat it as the tool you reach for when the simple strategies have demonstrably failed on your evaluation set, not as the default.

Whatever the strategy, one dial dominates: chunk size, and it pulls in two directions at once. Shrink the chunks and each vector represents one crisp idea — matching gets precise — but answers that span a boundary fragment, and each retrieved hit arrives stripped of the context around it. Grow the chunks and continuity survives, but the vector dilutes across topics (matching gets fuzzy), and fewer chunks fit the window, so one mediocre hit can crowd out two good ones. This is the same budget arithmetic as the context-engineering module, now applied to evidence: k chunks × chunk size is the retrieval slice of your token budget.

There is no universal number. The honest procedure is boring: pick a structure-appropriate strategy, a moderate size, and an overlap; then measure retrieval quality on your own questions and move the dial. The RAG End to End module builds that evaluation. A common middle path — retrieve by small chunk, then hand the model the surrounding section — shows up there too, under reranking and context assembly.

Interactive sorting exercise: Match each corpus to the chunking strategy you would reach for first.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.