Spending less per answer: tiers, batching, compression
Lesson 2 of 3 in GenAI Cost Engineering.
Once tokens are measured and attributed, optimization is three questions asked in order. Can the model read less? — prompt compression and context pruning. Can a cheaper model answer? — tiered FM usage. Can the work move off-peak? — batch inference and right-sized Provisioned Throughput. The professional exam rarely asks what a lever is; it describes a workload and asks which lever fits — and the wrong answers are real levers aimed at the wrong workload.
| Lever | What it attacks | The scenario tell | Watch out |
|---|---|---|---|
Prompt compression & context pruning | Input tokens per request | "The system prompt has grown"; "we send 25 chunks, answers cite 3" | Compress against an evaluation set — fidelity loss is silent until measured |
Response limiting | Output tokens per request (the pricier direction) | "Verbose answers"; "summaries keep running long" | A hard max-token cap can truncate mid-answer — pair it with concise-output instructions |
Tiered model usage | Price per token itself — most queries never see the flagship | "90% of questions are routine"; "mixed simple and complex traffic" | Validate the cheap tier against a golden set, or savings arrive as quality complaints |
Batch inference | Price per token for offline work, at the cost of latency | "Overnight"; "no user waiting"; "10 million records to S3" | Async only; no tool calling or structured output; not for provisioned models |
Provisioned Throughput right-sizing | The gap between committed model units and actual utilization | "Steady 24/7 volume"; "custom model in production"; "on-demand throttling" | Committing to peak locks in idle spend — commit to the floor, let on-demand absorb spikes |
Tiering deserves its own paragraph because AWS productized it. Intelligent prompt routing gives you a single serverless endpoint that predicts, per request, the response quality each model in a family would deliver, and routes to the cheapest one predicted to clear the bar — with a fallback model when prediction is uncertain. Build-your-own alternatives: a small classifier model grades complexity before invoking (cascading — try cheap first, escalate on low confidence), or distillation trains a small model on the flagship's outputs so the cheap tier is yours. Whichever you pick, the exam's measure of success is the cost-capability tradeoff: quality per query class against price per query class — measured with evaluations, not vibes.
Interactive sorting exercise: Which cost lever does each workload call for?
Key terms: Tiered model usage, Intelligent prompt routing (Bedrock), Batch inference (Bedrock), Prompt compression, Provisioned Throughput (Bedrock)
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.