Tokens, embeddings, and chunking for RAG
Lesson 4 of 5 in Transforming Data and Engineering Features.
Text has its own preparation pipeline, and MLA-C02 made it first-class. Three jobs: tokenization — models read subword tokens, not words, so token counts set context budgets and cost; embedding — an embedding model maps text (or images, with multimodal models) to vectors where similar meaning lands nearby; and, for retrieval-augmented generation, Chunking — splitting documents so that retrieval can return the right amount of context. Chunk too big and you stuff the prompt with noise and tokens; chunk too small and the answer's context is amputated.
Fixed-size + overlap
Cut every N tokens with 10–20% overlap. Predictable, cheap, token-budget-friendly — the default, and what Bedrock Knowledge Bases applies unless told otherwise.
Semantic
Split where meaning shifts, detected by comparing embeddings of neighboring sentences. Chunks are more coherent (fewer mid-thought cuts) at higher indexing cost. Pick it when fixed-size chunks keep slicing arguments in half.
Hierarchical
Parent–child: search matches small child chunks (precise), but the retriever returns the larger parent (context intact). The answer for long, structured documents — manuals, contracts — where the sentence that matches is not enough to answer with.
No chunking
Treat each document as one chunk. Right when documents are already retrieval-sized units — FAQ entries, product descriptions. (Bedrock Knowledge Bases offers all four strategies, plus fully custom chunking via a Lambda function.)
RAG tunes what the model sees; three other preparations change what the model is — and each wants differently shaped data.
| Goal | Data you prepare | Shape |
|---|---|---|
Fine-tuning (teach task behavior) | Labeled prompt–completion pairs | JSONL, one example per line; hundreds to thousands of high-quality pairs beat bulk |
Continued pre-training (absorb domain language) | Unlabeled domain corpus | Plain text at scale — contracts, clinical notes, internal wikis; no labels required |
Model distillation (smaller model, kept behavior) | Teacher outputs | Responses generated by the larger teacher model become the training set for the student |
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.