Token budgets, embedding compute, vector storage
Lesson 4 of 4 in Cost and Performance Engineering.
A GenAI bill has a different shape from an endpoint bill: on-demand foundation models meter tokens, in and out, per request — so cost scales with verbosity, not instance hours. That makes the levers linguistic as much as infrastructural. The single biggest one is model choice: a smaller model that clears the quality bar beats every micro-optimization on a larger one (and intelligent routing can send only the hard prompts to the big model). After that, attack the token stream itself.
Trim
Prompts bloat silently: system prompts accrete rules, few-shot examples multiply, RAG stuffs ever more chunks into context. Audit token counts per request (the GenAI observability dashboards chart them), cut dead instruction weight, retrieve fewer/better chunks, and set max output token limits — a truncation policy is both a cost cap and a latency cap. Output tokens are the expensive direction; "be concise" is a cost control.
Cache
When many requests share a long prefix — the same system prompt, the same document under discussion — prompt caching lets the model reuse the processed prefix instead of re-reading it every call, cutting both cost and latency on the repeated portion. The exam tell: "the first 90% of every prompt is identical".
Downsize
Evaluate smaller models against your real task before defaulting to the flagship (Bedrock evaluations make this measurable). Distill where quality allows. Route by difficulty so the expensive model only sees the prompts that need it.
Commit
Provisioned Throughput buys dedicated model capacity in model units, with commitment terms — the fit for steady, high-volume workloads where on-demand token pricing or throttling becomes the problem, and the required serving path for models you fine-tune in Bedrock. It is a throughput commitment: it changes how you pay, not how many tokens your prompts burn.
RAG adds two more meters. Embedding compute: every ingested chunk costs an embedding call, and re-embedding the corpus (new embedding model, new chunking) repeats the whole spend — batch it, and only re-embed what changed. Vector storage: each stored vector costs money monthly, and cost scales with vector count × dimensionality plus query volume. Chunking granularity is therefore a cost dial: halving chunk size roughly doubles vector count. Smaller-dimension embedding models, and index/storage tiers suited to your query latency needs, shrink the standing cost. A vector store full of stale documents is the GenAI version of the forgotten endpoint.
Horizontal bar chart of illustrative shares of monthly GenAI application spend: input tokens 38% (lever: trim prompts, cache the shared prefix, retrieve fewer chunks), output tokens 27% (lever: cap max output tokens), vector storage 15% (lever: coarser chunks, smaller embedding dimensions, prune stale documents), provisioned idle capacity 12% (lever: right-size committed model units), and embeddings compute 8% (lever: batch ingest and re-embed only what changed). Relative weights for one illustrative workload, not pricing data — the mix shifts with prompt length, traffic shape, and corpus size.
Key terms: Provisioned Throughput (Bedrock), AWS Cost Explorer, AWS Budgets, CloudWatch generative AI observability
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.