Where Each Meter Lives in the Stack
Lesson 3 of 3 in In Production: Where the Meters Run.
A production LLM system is a stack of five layers, and the meters are not spread evenly across it. Each meter is determined — set, mostly, and hard to change from anywhere else — at particular layers. Knowing which layer owns which meter is the difference between debugging in the right place and thrashing everywhere at once.
From the ground up: hardware (accelerators and the memory feeding them), the model (the Weights and architecture you chose), serving (the software that batches requests, manages the KV cache, and turns one model into many concurrent conversations), the adaptation layer (everything you wrap around the model — prompt templates, retrieved documents, few-shot examples, output filters), and your application (the product the user touches). Click each layer in the figure to see which meters are pinned to it and why.
A five-layer architecture stack, bottom to top: Hardware (the floor under cost and speed), Model (the quality ceiling), Serving (where latency and cost-per-token are manufactured), Adaptation layer (token volume and the security front line), and Your application (the latency users feel; who gets in). Each layer is annotated with the production meters — cost, latency, reliability, security — that are chiefly determined at that layer.
Run each meter through the stack and a pattern appears: every meter has one layer that owns most of it, plus one that surprises you.
Cost is a product of two numbers set at opposite ends: tokens-per-request (adaptation layer — your templates and retrieved context) times cost-per-token (serving and hardware). Teams instinctively look down the stack when the bill spikes; in practice the culprit is usually up at the adaptation layer, where someone added a document to every prompt.
Latency is manufactured at serving (queueing, batching, prefill) on hardware’s floor — but the adaptation layer quietly moves time-to-first-token, because every token it adds is more prefill before the first output token can exist. The app layer cannot make anything faster; it decides how the wait feels.
Reliability splits cleanly: the uptime half lives at serving (capacity, failover), the quality half at the model layer (version changes, drift) — and the app layer holds the mitigations (retries, fallbacks, degraded modes).
Security spans the three layers text passes through: who gets in (app), what goes in (adaptation — injection arrives inside retrieved documents and user messages), what comes out (model). Locking down only the app layer secures exactly one of the three doors.
The bill doubled overnight — where do I look first?
Adaptation layer. Diff what a request actually sends: prompt template changes, a new retrieval source, longer conversation history retained per turn. Token volume moves in big steps when someone edits this layer; cost-per-token (serving, hardware, provider pricing) rarely doubles silently.
Users say it got slower, but generation speed is unchanged — where do I look?
Two suspects: time-to-first-token grew because input tokens grew (adaptation layer — more prefill), or queueing at the serving layer under higher load. Tokens-per-second being flat is your clue that decode — and the hardware under it — is not the story.
Answers degraded but nothing in our code changed — where do I look?
Model layer first: did the provider update the model version, or did a deployment migrate? This is why version pinning matters where the platform offers it. Second suspect: the adaptation layer’s retrieval source changed what context the model sees.
The model revealed data it should not have — which door was open?
Trace all three text doors: did the app let the wrong user in (classic auth)? Did the adaptation layer put data in the Context window that this request never should have retrieved? Or did the model repeat something from its instructions it was told to keep quiet? Most real incidents are the middle door — retrieval scoping, not authentication.
In production
On managed clouds you can locate every layer of this stack — the difference between options from the last lesson is simply which layers the provider operates for you.
AWS
Your application and adaptation layer are always yours — application code plus whatever orchestrates prompts and retrieval. On Amazon Bedrock, serving, model, and hardware are the provider’s layers: you read their meters (token bills, latency metrics, quotas) without operating them. Move to SageMaker or EC2 and the serving layer becomes your software on GPU capacity you size — the same stack, with the ownership line drawn lower.
Azure
In Azure AI Foundry the ownership line is visible per deployment type: serverless APIs put serving, model, and hardware behind the platform’s tokens-per-minute meter; provisioned deployments give you the serving layer’s capacity dial (you size throughput, you pay for it idle); AKS with GPU nodes hands you the serving software itself. The app and adaptation layers stay yours in every case — and so do the meters they own: token volume and the injection surface.
Google Cloud
Vertex AI endpoints occupy the serving layer — you pick the model and the capacity, Google runs the software and hardware under it, and Cloud Monitoring exposes the latency and error meters per endpoint. Take the same open-weights model to GKE and the serving layer becomes code you deploy and patch. Either way, the count-tokens API meters the layer that stays yours everywhere: the adaptation layer’s token volume.
That is the domain. You can define an Large language model (LLM) in one honest sentence, explain Next-token prediction and why it produces both fluency and Hallucination, place any model on seventy years of history, name what these systems can and cannot do, read the lifecycle map — and now you know the four meters that every one of those ideas eventually lands on in production.
You are ready for L-02. Everything in this module traced back to one unexplained unit: the token. Why is the bill per token? Why does the model generate one at a time? Why does the Context window fill the way it does? The next domain opens the machine and answers all of it — starting exactly where the meters start, with tokenization.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.