Almost Never — and the Exceptions

Lesson 1 of 3 in In Production: Should You Ever Pre-train?.

You have now seen everything a pre-training run is made of: the data pipeline, the objective, the scaling laws, the FLOPs budget, the parallelism, the precision tricks, the loss spikes at 3 a.m. This closing module asks the only question that matters commercially: should your organization ever do any of this?

The honest answer, for almost everyone, is no — and the reason is not that pre-training is hard (though it is). It is that the alternatives are so much better than people expect. When a team says “we need our own model,” the need underneath is almost always one of three things: the model lacks knowledge (your documents, your data, last month’s facts), the model lacks behavior (your format, your tone, your domain’s conventions), or the model lacks skill (it genuinely cannot do the task). Each has a remedy that costs orders of magnitude less than a training run.

Knowledge gaps are what retrieval is for: fetch the relevant documents at request time and put them in the Context window. No training of any kind touches this problem better — pre-training bakes knowledge into Weights with a Knowledge cutoff; retrieval keeps it fresh and auditable. Behavior gaps yield to prompting first (instructions and examples in context) and to fine-tuning when prompting plateaus — supervised tuning on thousands of examples, not trillions of tokens. Skill gaps usually mean you tested the wrong model or the wrong prompt — and if the skill truly is not in any frontier model, thousands of your examples will rarely teach it either.

Should you pre-train? Walk it honestly

Interactive decision tree — outcomes:

  • Adapt an existing model

    Prompting, retrieval, fine-tuning — in that order, with evals between rungs. Honest cost: application-team time, per-token or per-hour serving, days to weeks. This is the right exit for almost every business need, including most that arrive labeled “we need our own model.”

  • Continued pre-training on an open-weights base

    Keep training a Base model on your corpus (mixed with general data) with the same next-token objective. Honest cost: a real GPU cluster for days to weeks, a data pipeline, distributed-training skills, and evals to catch forgetting — a middle path, not a shortcut.

  • Pre-train from scratch

    Everything this domain taught, at full price: a trillion-token-class corpus, thousands of accelerators for weeks to months, a training team, and an evaluation program. Honest cost: one of the largest engineering purchases your organization can make — justified for a handful of organizations on earth.

Look at what survives the tree. Novel domain language at scale — where your data is not English-about-your-industry but a different distribution altogether, and you hold enough of it to matter. Data sovereignty that reaches the weights — rare, because most regulatory needs are met by where data is processed, not by who trained the starting checkpoint; national and sector initiatives that must own the full provenance of a model are the exception. Model as the product — you are a lab, and the run is the business. Everything else exits to adaptation.

The middle exit deserves its own paragraph, because it is the one serious option between “prompt it” and “build a lab.” Continued pre-training takes an open-weights Base model and simply keeps going: same next-token objective, your corpus as the new Data mixture. You inherit everything the original run paid for — general language, reasoning, world knowledge bought with trillions of tokens — and spend your budget only on the delta. But it is still training: you need the data pipeline this domain opened with (Deduplication included), a cluster with real interconnect, a Learning rate schedule chosen for resumption rather than a fresh start, and evals that catch catastrophic forgetting — the base model’s general skills eroding as your narrow corpus overwrites them. The standard defense is mixing general data back in alongside your domain corpus, which means the mixture question returns even here. And the result is still a base model: it completes text. Making it usable takes the post-training pipeline of the next domain, whether you pre-trained from scratch or continued.

In production

The decision framework is visible in the shape of the platforms themselves: all three clouds productize the adaptation ladder as managed services, and full training infrastructure sits deliberately apart as a different tier of responsibility.

AWS

The managed path is Amazon Bedrock: hosted models invoked per token, with retrieval and fine-tuning-style customization offered as managed features — the whole adaptation ladder without owning an accelerator. Training from scratch or continued pre-training at scale means moving to SageMaker training infrastructure, where you own the data pipeline, the job configuration, and the convergence. The platform boundary is the decision tree drawn in services: crossing from Bedrock to a training cluster is crossing from “use a model” to “make a model.”

Azure

Azure AI Foundry carries the adaptation path: a model catalog, retrieval patterns, and managed fine-tuning against hosted models. Serious training runs live in Azure Machine Learning as distributed GPU jobs you specify and operate. The organizational tell is the same on every cloud: adaptation is an application-team activity inside the platform’s guardrails; pre-training makes you the operator of a supercomputer tenancy, with quota negotiations and capacity planning to match.

Google Cloud

Vertex AI hosts the ladder — model garden, retrieval tooling, managed tuning — while full-scale training runs on Cloud TPU or GPU infrastructure with your own training stack. Google’s own framing mirrors the tree: tuning services are sized for thousands of examples; TPU pod-scale training is sized for corpora with trillions of tokens. If your data fits in the first bucket, the second is the wrong purchase.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.