The Composite Answer

Lesson 3 of 3 in In Production: Choosing Your Lever.

The ladder is a diagnostic order, not a menu you pick one item from. Mature systems usually end up stacking levers, because each one is cheapest at a different job — and a shape you will meet again and again in production is: a System prompt carrying policy and task framing, Retrieval supplying facts, and sometimes a light Adapter locking in format or domain behavior. Not every system needs all three, and the stack should be earned rung by rung — but when you open up a serious deployed assistant, some version of this composite is what you tend to find.

The division of labor follows the matrix from lesson one. The system prompt holds what engineers change deliberately: policy, tone, the shape of the task. The index holds what changes daily and must be citable: the facts, kept fresh by an ingest pipeline and delivered per query as Grounding context. The weights hold what should never vary call to call and is too expensive to re-demonstrate each time: output format, domain register, the reflexes of the task.

One request through a composite system — each stage is a lever

  1. User query

    The only part of the prompt no lever controls — everything after this is your design.

  2. Retrieve

    Lever: retrieval (knowledge). Embed the query, search the index, rerank the candidates. The passages that survive become the facts the answer is allowed to use — refreshed by ingest, never by retraining.

  3. Assemble context

    Lever: context (policy and framing). The system prompt — rules, tone, refusal policy, output contract — plus the retrieved passages and the question, ordered static-first so caching can discount the stable prefix.

  4. Generate: base model + adapter

    Lever: weights (behavior). A base model, optionally with a LoRA adapter that locks in format and domain register — the patterns too subtle or too constant to spend context tokens re-demonstrating on every call.

  5. Grounded answer with citations

    Each claim traceable to a passage. A sample of these answers feeds the eval harness — the gate every lever change must pass before it ships.

A composite grounded-answer system prompt
You are {{ASSISTANT_ROLE}} for {{PRODUCT_NAME}}.

Rules:
- Answer ONLY from the context passages below. If they do not contain
  the answer, say "I don't have that information" — do not guess.
- Cite the passage id for every factual claim, like [doc-3].
- Respond in {{OUTPUT_FORMAT}}.
- {{TONE_AND_POLICY_RULES}}

Context passages:
{{RETRIEVED_PASSAGES}}

User question:
{{USER_QUESTION}}

The prompt-level half of a composite system: retrieval supplies the facts, and this template forbids inventing any others — the refusal rule is grounding enforced in context. Keep the static parts (role, rules) first and byte-stable so prefix caching discounts them; the retrieved passages and the question go last because they change on every call.

A composite is never finished, because the ground it stands on moves. Every lever choice in this domain was made against a model generation and a price sheet, and both change under you. A new model may follow instructions the old one needed an adapter for — deleting a lever from your stack. A longer or cheaper Context window can make deeper retrieval affordable, or make a summarize-then-retrieve workaround unnecessary. A price change flips a break-even you computed six months ago. Model deprecations force the issue whether you like it or not.

So treat every lever decision as a decision with a review date, not architecture carved in stone — and keep the exits cheap: the prompt in version control, the index rebuildable from source documents, the fine-tuning data and eval pipeline reproducible end to end. Teams that keep the exits cheap re-decide in a sprint; teams that don’t are still paying rent on last year’s answer.

In production

The composite is what the platforms are built for: their pieces are designed to stack in a single invocation, and the operational question becomes where each lever’s configuration lives and how a change to one is tested without disturbing the others.

AWS

A single Amazon Bedrock request can carry the whole stack: a versioned system prompt from prompt management, passages retrieved from a Knowledge Base, and a customized model as the invocation target — with guardrails wrapped around the result. The operational discipline the platform cannot supply is separation of change: version the prompt, the index snapshot, and the model artifact independently, change one at a time, and run the same eval suite against every change regardless of which lever it touched.

Azure

Azure AI Foundry composes the same triple — AI Search grounding, a fine-tuned deployment, and project-versioned prompt assets — and builds evaluation runs into the project workflow, which makes the gate from this lesson concrete: candidate configuration versus baseline on a fixed test set, before rollout. Treat prompt, index, and model version as three independently versioned deployment inputs; a rollback should be possible for each without touching the other two.

Google Cloud

Vertex AI stacks grounding services, tuned model endpoints, and managed prompts, with evaluation tooling to score a candidate stack against a baseline. One boundary line worth drawing for your team: when a model decides for itself whether to search — agentic retrieval — the orchestration logic belongs to agent frameworks (our sister AI Agent Academy’s territory); the lever composition, its cost shapes, and its eval gates are exactly what you learned here, unchanged.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.