Contextual vs Static Embeddings
Lesson 3 of 4 in Embeddings: Tokens Become Vectors.
The embedding layer has an obvious blind spot: it hands “bank” the same row in “we walked along the river bank” and “I opened a bank account.” One vector per token, context be damned. If that were the model’s final word on meaning, LLMs couldn’t function.
The rest of the stack is the fix. As the vector at each position flows upward through the Residual stream, Self-attention mixes in information from surrounding tokens and feed-forward layers rework it. A few layers in, the vector at the “bank” position is no longer the embedding row — it’s a Hidden state: a context-shaped representation in which the river reading and the finance reading have drifted far apart. Static at the entrance, contextual everywhere inside.
That contextual turn was a landmark in NLP — ELMo (Peters et al., 2018) showed that context-dependent vectors beat static ones across a range of tasks, and every modern Large language model (LLM) is built on the idea.
Bar chart with four bars. The hidden state of bank in the phrase river bank scores 0.78 against shore and 0.22 against deposit. The hidden state of bank in the phrase bank account scores 0.81 against deposit and 0.15 against shore.
This is also where a chronic terminology muddle lives. Engineers use the word “embedding” for three different objects, and mixing them up causes real design mistakes — like trying to build semantic search out of a model’s input rows. Keep the three apart:
| Kind | What it is | Context-aware? | What it’s for |
|---|---|---|---|
Embedding-layer row | One learned vector per vocabulary token — the lookup this module opened with | No — same vector in every sentence | The model’s own input step; not designed for anything else |
Hidden state | A position’s vector after attention and feed-forward layers have reshaped it | Yes — changes with every context | The model’s internal working representation; what interpretability tools probe |
Embedding-model output | One vector for a whole sentence or document, from a separate model trained specifically so that similar texts land close together | Yes — the whole input shapes it | Retrieval: semantic search, clustering, deduplication, RAG indexes |
In production
Inside the model, the embedding layer is a rounding error on your bill. Where embedding dimensionality really bites is retrieval infrastructure: every chunk you index becomes a d-dimensional vector, so d is a dial that scales your storage, memory, and query cost.
AWS
Vector search on AWS — Amazon OpenSearch Service vector engine, or pgvector on Amazon Aurora/RDS — keeps index structures in memory for fast approximate nearest-neighbour search. Memory grows roughly linearly with vector count × dimension, so doubling embedding dimension roughly doubles the index footprint and pushes you toward larger, costlier nodes before you’ve added a single document.
Azure
In Azure AI Search, vector fields count against index storage quotas, and dimension multiplies the size of every stored vector. Higher-dimensional embeddings mean fewer documents per partition and more partitions for the same corpus — dimension choice is effectively a capacity-planning decision, made before ingestion because re-embedding a corpus means paying the pipeline again.
Google Cloud
Vertex AI Vector Search prices along index size and serving capacity, both of which scale with dimension × vector count. The mechanism to remember: retrieval quality usually improves sub-linearly with dimension while cost grows linearly — so benchmark a smaller-dimension embedding model on your own retrieval task before defaulting to the largest one.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.