Embedding Models
Lesson 2 of 4 in RAG Mechanics: Embeddings and Search.
Where do these vectors come from? From a transformer — but one trained for a different job than your chat model. The standard design is the bi-encoder, two encoder “towers” (in practice usually the same weights used twice): one encodes the query, one encodes a passage, each producing a single vector, and relevance is scored as the similarity between the two — a cosine or dot product, one arithmetic operation instead of a model call.
That last clause is the entire economic argument. The alternative — feed query and passage into one model together and let Attention read them jointly — is called a Cross-encoder, and it judges pairs more accurately, because the model can line up “can’t log in” against “authentication failure” token by token. But a cross-encoder gives you no vectors to store: every query–passage score requires a fresh forward pass, so scoring a million passages means a million model calls per query. Sentence-BERT (Reimers & Gurevych, 2019) made the bi-encoder respectable by showing how to fine-tune encoders in a siamese setup so that plain cosine similarity between their pooled output vectors tracks semantic relatedness. Precompute the corpus side once, and search becomes geometry. The two designs are not rivals so much as stages: bi-encoder to search millions, cross-encoder to re-judge the top handful — that division of labor is the next module’s reranking lesson.
One confusion to kill early: your chat model already contains embeddings, and they are not these. The embedding matrix inside a generative Large language model (LLM) maps each token ID to a vector as the first step of the forward pass — machinery for generation, covered in the Inside the Transformer domain. An embedding model is a separate, complete model that outputs one vector per text and is trained so that nearness means relatedness. You will typically pair an embedding model from one family with a chat model from another, and nothing breaks — the vectors never enter the chat model; only retrieved text does.
Line chart drawing four vectors as arrows from the origin in a two-dimensional plane: a query vector for “how do I reset a user’s password?”, a document vector for a password-reset runbook chunk pointing in almost the same direction, a vector for an SSO outage postmortem at a wider angle, and a vector for an office plant watering rota pointing nearly perpendicular to the query. The small angle between query and runbook illustrates high cosine similarity.
Embedding models differ in the languages and domains they were trained on, in how long an input they accept — one reason Chunking exists at all, next lesson — and, most visibly, in output dimensionality. More dimensions give the vector more room to encode distinctions, but every dimension is a number you must store, move, and multiply for every chunk, forever. The vector store’s memory bill is a simple mechanism: dimensions × rows × bytes per number. Double the dimensions and you double the storage and the per-comparison compute across the whole index; quadruple the corpus and the bill quadruples with it. A slightly better model that doubles your vector width is not free — it is a standing tax on every query.
How do you pick one? MTEB, the Massive Text Embedding Benchmark, is the standard leaderboard for shortlisting candidates across retrieval and other embedding tasks — use it as a directory, not a verdict. Rankings on public benchmarks reorder easily on your corpus, your query style, your languages. The exam that matters is a small retrieval evaluation on your own data, which the RAG End to End module shows you how to build.
| Vector dimensions | Bytes per vector (fp32) | Per 1M chunks | Per 100M chunks |
|---|---|---|---|
384 | 1,536 B | ≈ 1.5 GB | ≈ 154 GB |
768 | 3,072 B | ≈ 3.1 GB | ≈ 307 GB |
1,536 | 6,144 B | ≈ 6.1 GB | ≈ 614 GB |
3,072 | 12,288 B | ≈ 12.3 GB | ≈ 1,229 GB |
Why cosine? Angles, dot products, and normalization
Similarity between two vectors a and b can be measured at least three ways: cosine similarity (the cosine of the angle between them), the dot product (a·b), and Euclidean distance (‖a − b‖). Cosine is the usual default for text because it ignores vector length and compares only direction — and length is where nuisance variation tends to accumulate, while the meaning models are trained to encode lives in direction.
Normalization ties the three together. If you scale every vector to unit length, then a·b is the cosine, and Euclidean distance becomes a monotone function of it: ‖a − b‖² = 2 − 2·(a·b) for unit vectors. Ranked retrieval only cares about order, so on normalized vectors, all three metrics return the same ranking — which is why many stacks store normalized vectors and use fast inner-product search underneath, whatever “metric” the configuration screen says.
Two practical edges. First, some models are trained so that raw dot product (unnormalized) is the intended score — length then carries real signal, and normalizing it away changes rankings. Use the metric the model’s documentation specifies. Second, the index and the model must agree: an index built for Euclidean distance over unnormalized vectors will happily return neighbors that a cosine scoring would order differently. Metric mismatch is a quiet, entirely preventable retrieval bug — check it once at setup and pin it next to the model version.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.