Managed API or Your Own GPUs — a First Look

Lesson 2 of 3 in In Production: Where the Meters Run.

Before any code is written, every team that ships an LLM feature faces the same fork: call a model someone else runs, or run one yourself. You saw in the lifecycle module that models reach you as APIs, platforms, or downloadable weights — this lesson is about choosing between them, and about being honest that the choice is really three options, not two.

Managed API. You send requests over HTTPS to a model hosted by its provider and pay per token. You never see a GPU. Fastest start, least control: the provider chooses the hardware, the serving software, the update schedule, and — eventually — when your model version is retired.

Managed platform. A cloud platform runs the serving infrastructure for a model you select — often including Open weights models — inside your cloud environment. You choose and pin the model, set capacity, keep traffic inside your account boundary; the platform owns the servers, the serving stack, and the patching.

Self-hosted. You take open weights, rent or buy GPU capacity, run a serving stack, and own everything: which model, which Weights version, which optimizations, what happens to every byte of every prompt. Also: capacity planning, on-call, security patching, and the bill for idle GPUs at 3 a.m.

The classic framing applies cleanly here: how much of this is your differentiator, and how much is undifferentiated heavy lifting — work that must be done well but that no customer will ever choose you for? Serving infrastructure is heavy lifting for almost everyone. It stops being “undifferentiated” only when control itself is the requirement: data that may not leave your infrastructure, a custom model nobody else hosts, or traffic so large and steady that owning the serving layer changes the economics.

The first-look hosting decision

Interactive decision tree — outcomes:

  • Managed API

    Start here — most teams should. Fastest iteration, zero serving operations, per-token pricing that scales to zero when unused. The honest trade: you inherit the provider’s rate limits, model deprecation schedule, and update cadence, and your unit economics are set by someone else’s price list. Revisit the decision when scale, control, or model needs change — it is a door, not a wall.

  • Managed platform

    The middle path: model choice (including open weights) and account-boundary control, without owning servers. You can pin model versions and keep traffic inside your environment. The honest trade: you still depend on the platform’s catalog, capacity, and feature set, you pay for provisioned capacity whether or not you use it, and more configuration surface is yours to get wrong.

  • Self-hosted

    Maximum control: your weights, your serving stack, your data path, your economics. The honest trade: you now run a distributed GPU system — capacity planning, batching and caching software, security patching, monitoring, and on-call are all yours, and the meters you inherit from providers elsewhere must be built here from scratch. Choose this because control or measured economics demand it, not because it sounds cheaper.

Notice what the tree does not ask: “which option is best?” There is no best — there is a fit between your constraints and each option’s honest trade. Two warnings from teams who have walked it:

Self-hosting is rarely cheaper than it looks. The comparison people run is per-token API price versus GPU rental price, and it flatters self-hosting because it omits the serving software, the engineers, the idle capacity you keep for traffic spikes, and the evaluation work to confirm your deployment matches the quality you had before. Run the comparison with all of it included, on measured traffic, before you move.

Managed is not zero-risk. Providers deprecate model versions, change quotas, and update behavior; your product inherits every one of those decisions. The mitigation is not self-hosting — it is treating the provider like any dependency: pin versions where the platform allows it, test before migrating, and keep your prompts and evals portable so switching costs stay finite.

In production

Whichever branch of the tree you land on, you inherit a bundle of mechanisms — the difference is which bundle. All three clouds offer all three options.

AWS

The managed-API path (Amazon Bedrock invoking hosted models) hands you quotas, per-model token pricing, and the provider’s model lifecycle. The platform path (Bedrock with provisioned capacity, or SageMaker hosting an open-weights model) hands you capacity you must size and pay for while idle, in exchange for version pinning and account-boundary control. Self-hosting on EC2 GPU capacity hands you the full serving stack: your batching software, your patching, your autoscaling.

Azure

Azure AI Foundry spans the same three rungs: serverless token-billed model APIs at one end, managed deployments with provisioned throughput and model-version pinning in the middle, and self-managed serving on GPU virtual machines or AKS at the far end. The mechanism to notice is quota as tokens-per-minute on the managed rungs — throughput is something you request and are granted, not something you build.

Google Cloud

Vertex AI’s Model Garden exposes the pattern directly: the same open-weights model can often be consumed as a pay-per-token API, deployed to a managed endpoint you size and control, or exported to run on your own GKE GPU infrastructure. Same weights, three positions of the control dial — which makes GCP a clean illustration that the decision is about operations and control, not about the model itself.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.