The RAG Stack
Lesson 2 of 4 in Reference Architectures.
The second architecture is the managed-API app plus a document pipeline: Retrieval-augmented generation (RAG). Structurally it is two pipelines sharing one index. The ingest lane turns source documents into searchable vectors; the serve lane answers queries by retrieving from that index and grounding generation on what it finds. They run at different tempos — serve runs per request in milliseconds, ingest runs when documents change — and most RAG failures come from forgetting they are separate systems with separate owners, schedules, and failure modes.
Trace both lanes in the diagram. The left column is ingest; the right column is serve; the refresh loop is what keeps them honest with each other.
The RAG stack: ingest lane, serve lane, refresh loop
- Source systems
Wikis, document stores, databases, tickets. Managed connectors exist — Bedrock Knowledge Bases documents connectors for Amazon S3, SharePoint, Confluence, Google Drive, OneDrive, and a Web Crawler; Vertex AI RAG Engine documents ingestion from local files, Cloud Storage, and Google Drive. Access control starts here: whatever enters the index can surface in an answer.
- Parse + chunk
Documents become retrievable units. Chunk size and boundaries decide what a Retrieval hit can ever contain — the highest-leverage, least-glamorous decision in the stack. Managed offerings handle this for you (Bedrock Knowledge Bases documents Smart Parsing across document types); own pipelines make it an explicit, versioned choice.
- Embed
Each chunk goes through an Embedding model to become a vector. The embedding model is a frozen dependency: change it and every stored vector must be recomputed, because old and new vectors do not live in the same space.
- Vector store
The Vector database holding vectors plus chunk text and metadata. In managed mode the platform hosts it; in customer-managed mode you bring your own (Bedrock documents Amazon OpenSearch Serverless, Aurora, and Neptune as options for its customer-managed type). Metadata filters — tenant, permission, date — are what make retrieval safe in multi-user systems.
- Refresh loop
Change detection and re-ingestion: scheduled syncs or event triggers re-parse and re-embed what changed. This loop is the difference between a knowledge system and a launch-day snapshot — and the piece most often left out of the design doc.
- User query
The serve lane begins as the managed-API app did: gateway, auth, input guardrails. Everything from lesson one still wraps this lane — RAG adds nodes, it removes none.
- Retrieve
The query is embedded and the store returns nearest chunks, filtered by metadata and permissions. Recall problems here are invisible downstream: the model cannot ground on a chunk retrieval never returned.
- Rerank
A second, more precise model reorders candidates so the best evidence fits the Context window budget. Documented in the managed offerings: Bedrock Knowledge Bases supports reranking models, and Azure OpenAI On Your Data describes retrieval, filtration and reranking before generation.
- Grounded generation
The model answers from retrieved chunks placed in the prompt — Grounding. Platform guardrails can check faithfulness: Bedrock Guardrails documents contextual grounding checks for detecting ungrounded RAG responses, and Azure’s optional filters include groundedness detection.
- Answer + citations
Citations point back to source chunks — Bedrock Knowledge Bases documents citation of retrieved sources. Citations are the user-facing contract of the whole stack and your best debugging tool: a wrong answer with citations tells you which lane failed.
Each cloud will sell you the middle of this diagram as a managed service, so the real design decision is which lane you keep. Handing over ingest buys connectors, parsing, and index operations; keeping it buys control over chunking, embeddings, and permissions — the levers that decide retrieval quality. The Adaptation domain teaches those levers; here is what the platforms name their versions of the box.
| AWS | Azure | Google Cloud | |
|---|---|---|---|
Managed piece | Amazon Bedrock Knowledge Bases — query it and it retrieves from your connected data to ground responses, with citations | Azure OpenAI On Your Data — files ingested (cracked, chunked, embedded) into an Azure AI Search index; grounded generation at inference | Vertex AI RAG Engine — documented as “a configurable managed RAG service” covering the full RAG lifecycle |
Coverage | Two documented types: Managed (Bedrock runs ingestion, indexing, storage, retrieval; multi-hop retrieval; reranking) and Customer-managed (your own pipeline over your own vector store) | Ingestion through grounded generation: intent generation, retrieval, filtration and reranking, then response — over sources including AI Search indexes, Blob Storage, file upload, URLs, and Cosmos DB’s integrated vector database | Ingestion, chunking, embedding, indexing into a corpus, retrieval, and generation; usable as a grounding source alongside Google Search and other options |
Status notes | Documented connectors: S3, SharePoint, Confluence, Google Drive, OneDrive, Web Crawler; supports multimodal (image) search | Deprecated — documented retirement October 14, 2026, with migration guidance toward Foundry Agent Service and Foundry IQ knowledge bases (agent-platform territory — see our sister AI Agent Academy) | Available in a documented region list at varying launch stages; not supported on the global endpoint |
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.