The Hidden Costs

Lesson 3 of 4 in Cost Modeling.

The meter you watch is not the bill you pay. Around every LLM workload sits a ring of adjacent charges — individually small, collectively material, and invisible in any per-token comparison.

Data transfer. Tokens are small; the data around them is not. Document ingestion pipelines, embedding syncs, model weights pulled between regions, logs shipped to an external observability stack — Egress between regions and out to the internet is metered on all three clouds, and multi-region architectures multiply it. (Cross-region inference is its own mechanism with its own pricing behavior — the domain’s capstone module returns to it.)

Storage. Model artifacts and checkpoints; every fine-tune you keep (Bedrock, for example, documents monthly storage charges for custom models); prompt and response logs retained for debugging and compliance; traces; and the Vector database behind RAG — an index that grows with your corpus and bills whether or not anyone queries it.

Where the total actually lives

  1. The meter you watch — Tokens · capacity-unit-hours · GPU-hours

    The line every dashboard tracks and every comparison spreadsheet models. Real — but a floor, not a total. Tier sizes here are illustrative: the point is the ordering, not the proportions.

  2. Adjacent metered lines — On the invoice, off the model

    Charges that follow the workload around: data transfer between regions and out to the internet, storage for artifacts, logs, and indexes, and premium networking such as private connectivity. Each is metered and visible — on someone else’s invoice page.

  3. Behavioral multipliers — Your own traffic, multiplied

    Every retry, fallback, over-long generation, eval run, and staging replay is metered exactly like production. The meter counts tokens, not useful tokens — the gap between Throughput and Goodput bills all the same.

  4. People — The biggest line has no meter

    Engineers running upgrades, on-call, capacity planning, quota management, evals, and cost reviews. It lands in payroll, never on the cloud invoice — and it is routinely the deciding line between managed and self-hosted.

The behavioral multipliers deserve arithmetic respect because they compound. A retry policy that allows two extra attempts turns a failing path into up to triple token spend for a single user answer. A model that pads its answers converts verbosity straight into the expensive output meter — max-token caps and stop sequences are cost controls, not just UX polish. Dev and staging replay production-shaped traffic at production token rates, and unlike production their volume scales with engineering activity, not with users. None of this is waste, exactly; unmodeled is the problem. Give each multiplier a named variable — a retry factor m_retry ≥ 1, an over-generation factor m_gen, an environment multiplier m_env — and your projections stop being systematically low.

The bill doubled but traffic didn’t

Look for multipliers, not volume: a retry or fallback chain activated by an incident, a prompt-template change that quietly inflated input tokens, or a model version with different verbosity. Diff tokens per request over time, not request counts.

Egress was supposed to be negligible

It is — until embeddings syncs, log shipping, or artifact pulls start crossing regions or leaving the cloud. Audit which components moved relative to the data they read, and price the paths that now cross a boundary.

Staging costs almost as much as production

Eval suites and load tests replay full traffic at full price. Route latency-tolerant suites to the documented batch/Flex tiers, cap generation length in tests, and sample instead of replaying everything nightly.

The vector index grew a bill of its own

Index storage and its serving compute scale with corpus size, not query volume — a RAG system’s cost floor rises as documents accumulate even if usage is flat. Model the index as its own line with its own growth rate.

We saved on tokens and lost on people

The self-hosted fleet beat the per-token bill, then consumed an engineer-year in upgrades, capacity firefighting, and on-call. C_ops was never in the spreadsheet. Put it there — at loaded cost, not salary.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.