The Four Meters
Lesson 1 of 3 in In Production: Where the Meters Run.
You now know what an Large language model (LLM) is, what Next-token prediction really means, where the limits are, and how a model moves through its lifecycle. This module closes the domain with the question every later domain answers in more depth: what happens when you actually run one?
The moment an LLM serves real traffic, four meters start running. They are the same four meters every production system has — but each one gets a twist that is specific to how LLMs work, and if you only know the classic version, the twist is exactly where you get surprised.
Cost. Plain version: what you pay to run the system. The LLM twist: the bill is denominated in tokens, not requests or hours — and it is asymmetric. Input tokens (everything you send: the system prompt, the conversation so far, retrieved documents) are metered separately from output tokens (everything the model generates), and output tokens are typically priced several times higher. A “short question” with a long prompt template behind it is not a short request.
Latency. Plain version: how long users wait. The LLM twist: there is no single number. The wait splits into time-to-first-token — how long before anything appears — and tokens-per-second — how fast text flows once it starts. A model can start answering almost instantly and still take thirty seconds to finish, or make you stare at a spinner and then finish in a blink. Users forgive slow finishes far more readily than slow starts, which is why streaming the answer token by token is the default UX everywhere.
Reliability. Plain version: is the service up, and does it return correct responses? The LLM twist: an LLM service can be 100% up — every request returns HTTP 200 in good time — while the answers quietly get worse. Model versions change, providers update systems around the model, Sampling makes outputs nondeterministic, and quality can drift on exactly the slice of traffic you care about. Uptime dashboards do not measure any of that. Reliability for LLMs means uptime plus sustained answer quality, and the second half needs its own instruments — that is what the Evaluation domain is for.
Security. Plain version: who can get in, and what can they reach? The LLM twist: the classic perimeter still matters (keys, roles, network), but a whole new surface opens on the output side. The model can be talked into ignoring its instructions by text hidden in a document it reads (prompt injection), it can leak data it was shown in its Context window, and it can state falsehoods with total confidence (Hallucination) that your product then presents as fact. None of these are break-ins. The requests are authenticated, the network is fine — the content is the attack surface.
| Meter | Everyday unit | The LLM twist | Where you will meet it again |
|---|---|---|---|
Cost | Money per request, per hour, per month | Metered per token, with input and output priced separately — and output tokens costing several times more | Inference & Serving (L-05) and LLMs on the Cloud (L-09) — plus Tokenization, next domain, where the counting unit is defined |
Latency | Milliseconds until the response | Splits in two: time-to-first-token (the wait before anything appears) and tokens-per-second (how fast it flows after) | Inference & Serving (L-05): batching, caching, and the serving tricks that move both numbers |
Reliability | Uptime, error rate | Includes quality drift: the service stays up while answers get worse — nondeterminism and model changes need their own monitoring | Evaluation (L-07): how to measure answer quality so drift is visible |
Security | Who gets in: auth, keys, network | Adds what the model says: prompt injection, data leakage, confident falsehoods — attacks that arrive as ordinary text | Security & Risk (L-08): the model-level risk map, threat by threat |
One more thing the matrix cannot show: the meters push against each other. Serving many requests together in a batch makes each token cheaper but can make each user wait longer. A bigger, better model improves answer quality and raises both cost and latency. Stricter output filtering improves the security meter and adds a processing step to the latency meter. Production LLM engineering is rarely “make this meter better” — it is “which meter am I willing to move, and by how much, to buy improvement on that one?” Every later domain in this academy is, in part, a course on one of these dials.
Why output tokens cost more: prefill vs decode
The asymmetric bill is not marketing — it mirrors the hardware. Serving a request has two phases with very different shapes.
Prefill processes your entire input at once. Because every input token is already known, the model can compute attention for all of them in parallel, in a single pass that keeps the accelerator’s arithmetic units saturated. This is efficient work, and it is what determines most of the time-to-first-token: nothing can be generated until the whole prompt has been read.
Decode generates output one token at a time, because each prediction depends on the token before it. Every step re-reads the stored attention state for everything so far — the KV cache — so each output token means another full trip through the model’s Weights and cache, and the step is limited by how fast memory can feed the chip rather than by raw arithmetic. Sequential, memory-bound work is expensive work. Tokens-per-second is simply the speed of this loop.
So the two line items on the bill are two different computations: input tokens buy parallel prefill, output tokens buy sequential decode. That is also why the two latency numbers move independently — a long prompt inflates time-to-first-token, a long answer inflates total generation time — and why serving systems (Inference & Serving, L-05) treat prefill and decode as separate problems to optimize.
In production
The four meters are not abstractions — managed LLM platforms expose each one as a concrete surface you can read on day one.
AWS
On Amazon Bedrock the cost meter is the token-metered bill (input and output counted separately per model); the latency meter surfaces through streaming responses and invocation-latency metrics in CloudWatch; reliability is service quotas, throttling behavior, and your own logged quality checks on model outputs; the security meter runs through IAM for who may invoke which model and Guardrails-style content controls for what responses may say.
Azure
Azure AI Foundry denominates both billing and rate limits in tokens — quota is tokens-per-minute, so cost and throughput are the same meter read two ways. Latency shows up as streamed responses and per-deployment metrics; reliability as deployment health plus your own evaluation of outputs across model-version updates; security as Entra ID and network isolation on the way in, and built-in content filtering on the way out.
Google Cloud
Vertex AI meters generative usage in tokens and provides a count-tokens API for pre-flight cost checks; latency and error metrics flow to Cloud Monitoring per endpoint; reliability spans endpoint availability plus model-version pinning so quality does not shift under you silently; security combines IAM on the request path with configurable safety filters on generated content.
Key terms: Token, Context window, KV cache, Hallucination, Open weights
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.