LLM Academy glossary

202 terms with primary sources.

Large language model (LLM) (Large Language Model)

A large language model is a neural network — today almost always a Transformer — trained on large text corpora to predict the next Token. Everything an LLM appears to do (answer, translate, write code) is produced by repeatedly sampling the next token from the probability distribution the model computes over its vocabulary.

Token

A token is the unit of text a model processes: usually a subword chunk (like token + ization) produced by a tokenizer such as Byte-pair encoding (BPE). Models never see words or characters directly — only sequences of token ids. Tokens are also the billing and capacity meter: context windows, prices, and latency are all counted in tokens.

Next-token prediction

Next-token prediction is the objective LLMs are trained on: given a sequence of Tokens, output a probability distribution over the whole Vocabulary for what comes next. Training minimizes the surprise (cross-entropy) of the true next token. That one objective, at sufficient scale, produces the capabilities we call an LLM.

Transformer

The transformer is the architecture introduced by Vaswani et al. (2017) in "Attention Is All You Need". It processes all tokens in parallel through stacked blocks, each combining Self-attention (tokens exchanging information) with a Feed-forward network (FFN) (per-token computation), joined by Residual stream connections and normalization.

Vaswani et al. 2017, arXiv:1706.03762

Parameter

A parameter (or weight) is a single learned number in the network — an entry in an embedding matrix, an attention projection, or a feed-forward layer. Model names like "7B" count parameters in billions. Parameter count sets the memory footprint of the weights and, together with architecture, the compute per token.

Weights

Weights are the complete set of learned Parameters that define a trained model. "Open weights" means this artifact is published; running a model yourself means loading these numbers (at some precision) into accelerator memory.

Context window

The context window is the maximum token count a model can process in a single sequence — prompt and response together. Nothing outside it exists for the model. Window size is set by training and position-encoding choices, and attending over long contexts has real memory and latency costs.

Vocabulary

The vocabulary is the fixed set of Tokens a tokenizer can produce and a model can emit — typically tens to hundreds of thousands of entries. The model’s final layer scores every vocabulary entry at every step; that is why vocabulary size shapes both the embedding and output layers.

Logits

Logits are the raw scores the model outputs for each Vocabulary entry at a given step. A softmax converts them into a probability distribution; sampling settings like temperature reshape that distribution before a token is drawn.

Embedding

An embedding is a learned vector of numbers standing in for a token, word, or passage. Inside an LLM, an embedding matrix turns token ids into vectors; separately, dedicated embedding models map whole texts into spaces where distance tracks semantic similarity — the machinery behind vector search and RAG.

Inference

Inference is using a trained model: feeding tokens in, getting next-token distributions out, and sampling a response. It has its own engineering discipline — batching, caching, quantization — because inference, not training, is where most production cost and latency lives.

Sampling

Sampling is how a single next token is chosen from the probability distribution the model computes. Greedy decoding always takes the top token; temperature, top-k, and top-p reshape or truncate the distribution first. Sampling is why the same prompt can produce different answers.

Softmax

The softmax function exponentiates and normalizes a vector of raw scores (Logits) into probabilities that sum to 1. It appears twice in a transformer: over attention scores inside every head, and over the vocabulary at the output.

Attention

Attention lets a model decide, for each position, which other positions to draw information from — and how strongly. Instead of compressing everything into one fixed summary, each token computes weighted mixtures over the whole sequence. Attention predates the transformer (it began in machine translation), but the transformer made it the only way tokens communicate, via Self-attention.

Vaswani et al. 2017, arXiv:1706.03762

Self-attention

Self-attention is Attention applied within one sequence: every token builds queries, keys, and values from the same input and gathers context from its neighbors. It is the transformer’s communication step — the only place information moves between positions, with the Feed-forward network (FFN) doing per-position work in between. In LLMs it is causal: Causal masking stops tokens from looking ahead.

Vaswani et al. 2017, arXiv:1706.03762

Query, key, value (Q/K/V)

In Self-attention, each token is projected into three roles: a query (what this position is looking for), a key (how this position advertises itself), and a value (the information it contributes if selected). Query–key dot products produce Attention scores; after a Softmax, those weights mix the values into the token’s new representation. Keys and values are what the KV cache stores during generation.

Vaswani et al. 2017, arXiv:1706.03762

Attention head

An attention head is a single self-attention unit with its own Query, key, value (Q/K/V) projections, operating on a slice of the model’s width (Head dimension). Each head can learn a different relationship — tracking syntax, copying names, matching brackets. A layer runs many heads side by side as Multi-head attention, and interpretability work has found heads with recognizable jobs, like the Induction head.

Vaswani et al. 2017, arXiv:1706.03762

Multi-head attention

Multi-head attention splits attention into many parallel Attention heads, each with its own smaller projections, then concatenates and re-projects their outputs. The point is diversity: one softmax-weighted average can only express one mixing pattern per token, but many heads let a layer attend to several kinds of relationships at once. Variants like Multi-query attention (MQA) and Grouped-query attention (GQA) keep many query heads while sharing keys and values to shrink the KV cache.

Vaswani et al. 2017, arXiv:1706.03762

Multi-query attention (MQA) (Multi-Query Attention)

Multi-query attention keeps many query heads but shares a single key and value head across all of them. That slashes the size of the KV cache and the memory traffic that dominates generation speed, at some cost in quality compared to full Multi-head attention. Grouped-query attention (GQA) later offered a middle ground between the two.

Shazeer 2019, arXiv:1911.02150

Grouped-query attention (GQA) (Grouped-Query Attention)

Grouped-query attention divides the query heads into groups, and each group shares one key/value head. It interpolates between full Multi-head attention (every head has its own K/V) and Multi-query attention (MQA) (all heads share one), recovering most of the quality while keeping the KV cache small. It has become a default choice in modern open-weight LLMs.

Ainslie et al. 2023, arXiv:2305.13245

KV cache

The KV cache stores every previous token’s keys and values (from Query, key, value (Q/K/V) projections) so that generating each new token only requires computing its query against the cached past. Without it, each step would redo attention math for the whole prefix. The cache grows with sequence length and often becomes the memory bottleneck of serving — which is why architectures like Grouped-query attention (GQA) and Multi-query attention (MQA) exist to shrink it.

Feed-forward network (FFN)

The feed-forward network is the second half of every transformer block: each token’s vector is expanded to a wider dimension, passed through an Activation function, and projected back down — with no communication between positions. If Self-attention is where tokens talk, the FFN is where each token thinks alone; it holds the majority of a dense model’s parameters and is widely viewed as where much of the model’s factual knowledge lives. In Mixture of experts (MoE) models, the FFN is what gets replaced by a set of Experts.

Vaswani et al. 2017, arXiv:1706.03762

Activation function

An activation function is the nonlinear step applied inside a layer, such as ReLU, GELU, or the gated SwiGLU. Without it, any stack of linear layers would collapse into a single linear transformation, no matter how deep. In transformers it sits in the middle of the Feed-forward network (FFN), between the expand and contract projections.

SwiGLU

SwiGLU is a gated Activation function variant used in the Feed-forward network (FFN) of most recent LLMs (Llama, PaLM, and many others). Instead of one expanded projection pushed through a nonlinearity, it computes two: one carries content, the other — passed through a Swish nonlinearity — acts as a gate that scales it elementwise. In the original evaluation it outperformed plain ReLU/GELU feed-forward layers at matched compute.

Shazeer 2020, arXiv:2002.05202

Residual stream

The residual stream is the persistent vector (width d_model (model dimension)) each token carries from embedding to output. Every Self-attention and Feed-forward network (FFN) layer reads from the stream and adds its result back, rather than replacing it — so the stream acts like a shared communication bus that layers write updates onto. The framing comes from interpretability work; mechanically, these are the residual (skip) connections that also make very deep networks trainable.

LayerNorm

Layer normalization rescales each token’s vector — subtracting its mean and dividing by its spread, then applying a learned gain and bias — so activations stay in a stable range no matter how deep the network gets. In transformers it brackets every attention and Feed-forward network (FFN) sublayer; where exactly it sits is the Pre-norm vs post-norm choice. Many recent LLMs swap it for the simpler RMSNorm.

Ba et al. 2016, arXiv:1607.06450

RMSNorm

RMSNorm simplifies LayerNorm by dropping the mean subtraction and bias: it rescales each token’s vector by its root-mean-square magnitude and applies a learned gain. The observation was that the re-centering step contributes little — normalizing magnitude alone works about as well and costs less. It is the normalization used in Llama-family and most other recent open-weight LLMs.

Zhang & Sennrich 2019, arXiv:1910.07467

Pre-norm

Pre-norm places the normalization (LayerNorm or RMSNorm) before each attention and Feed-forward network (FFN) sublayer, normalizing what the sublayer reads while leaving the Residual stream itself untouched. The original transformer normalized after the residual addition (post-norm), which turns out to be harder to train at depth without careful warmup. Pre-norm’s cleaner gradient path is why essentially all modern LLMs use it.

Positional encoding

Positional encoding is how order gets into a transformer: Self-attention is symmetric over positions, so without it "dog bites man" and "man bites dog" would look identical. The original transformer added Sinusoidal encoding vectors to the input embeddings; modern LLMs mostly rotate queries and keys with Rotary position embedding (RoPE) instead. The choice of positional scheme also shapes how far a model’s usable Context window can stretch.

Vaswani et al. 2017, arXiv:1706.03762

Rotary position embedding (RoPE) (Rotary Position Embedding)

RoPE encodes position by rotating each query and key vector by an angle proportional to its position in the sequence. Because a dot product between two rotated vectors depends only on the difference of their angles, attention naturally sees relative distance rather than absolute position. It is the Positional encoding used by most modern LLMs, and stretching its rotation frequencies is the basis of most Context extension tricks.

Su et al. 2021, arXiv:2104.09864

Sinusoidal encoding

Sinusoidal encoding is the original transformer’s Positional encoding: each position gets a fixed vector built from sine and cosine waves at a spectrum of frequencies, added to the token’s Embedding. Fast-oscillating dimensions distinguish neighbors; slow ones distinguish distant regions — like a clock with many hands. Modern LLMs have largely replaced it with Rotary position embedding (RoPE), but the frequency-spectrum idea carries straight over.

Vaswani et al. 2017, arXiv:1706.03762

Context extension

Context extension covers the tricks for pushing a model past its trained sequence length — most commonly by rescaling Rotary position embedding (RoPE) frequencies (position interpolation and its refinements) so long sequences map into the positional range the model already understands, often followed by a short fine-tune on long documents. Extension is not free: attention cost and KV cache memory still grow with length, and models can attend poorly over regions far beyond what training covered.

Byte-pair encoding (BPE) (Byte-Pair Encoding)

Byte-pair encoding builds a Subword vocabulary by starting from individual characters and repeatedly merging the most frequent adjacent pair into a new token, until the Vocabulary reaches its target size. Frequent words end up as single tokens; rare words split into pieces; nothing is ever out-of-vocabulary. Originally a compression algorithm, it was adapted for translation and is now — usually in its Byte-level BPE form — the dominant way LLM tokenizers are trained.

Sennrich et al. 2015, arXiv:1508.07909

WordPiece

WordPiece is a Subword tokenization algorithm best known from BERT: like Byte-pair encoding (BPE) it grows a vocabulary by merging pieces, but it chooses the merge that most improves a language-model likelihood over the training corpus, rather than the most frequent pair. Its visible signature is the ## prefix marking pieces that continue a word. It mattered historically, though new LLMs mostly use Byte-level BPE or Unigram tokenization.

Unigram tokenization

Unigram tokenization works top-down, the opposite direction from Byte-pair encoding (BPE): start with a large candidate vocabulary of Subword pieces, treat text as generated by picking pieces independently, and iteratively prune the pieces whose removal hurts corpus likelihood least. Because many segmentations of a word remain possible, it can also sample alternative tokenizations during training (subword regularization). It ships as an option in the SentencePiece library.

Kudo 2018, arXiv:1804.10959

Byte-level BPE

Byte-level BPE runs the Byte-pair encoding (BPE) merge procedure over raw bytes rather than Unicode characters. Since every possible input is a byte sequence, nothing can ever be out-of-vocabulary — no unknown-token fallback needed for rare scripts, emoji, or corrupted text. Popularized by GPT-2, it is the scheme behind most frontier-model tokenizers today; the trade-off is that text in underrepresented languages can shatter into many bytes, costing more Tokens for the same content.

Subword

A subword is a text unit smaller than a word but usually bigger than a character — token + ization. Word-level vocabularies explode and still miss rare words; character-level ones make sequences painfully long. Subwords are the compromise every modern Tokenizer makes: common words stay whole, rare words decompose into reusable pieces, and the Vocabulary stays a fixed, manageable size.

Sennrich et al. 2015, arXiv:1508.07909

Tokenizer

The tokenizer converts raw text into a sequence of Token ids and back again. It is built before the model — by an algorithm like Byte-pair encoding (BPE), WordPiece, or Unigram tokenization — and then frozen; the model never sees text, only the ids the tokenizer produces. Its quirks leak everywhere: how numbers split, why some words cost more tokens than others, and many classic failure cases (counting letters, spelling backwards) trace back to tokenization.

Mixture of experts (MoE) (Mixture of Experts)

A mixture-of-experts model replaces each block’s single Feed-forward network (FFN) with many parallel Expert networks plus a Router that sends each token to only a few of them. That decouples total parameter count from per-token compute: the model can be enormous while each token activates only a small slice (Sparse activation). Many frontier models are MoEs; the price is more memory to hold all experts and training care to keep them evenly used.

Shazeer et al. 2017, arXiv:1701.06538

Expert

An expert is one of the parallel Feed-forward network (FFN) copies inside a Mixture of experts (MoE) layer. The Router picks a small subset of experts for each token, so any single expert only sees a fraction of the traffic. Despite the name, experts rarely map to clean human topics — the specializations that emerge tend to be about token patterns and syntax more than subjects like “biology”.

Router

The router (or gating network) is the small learned layer in a Mixture of experts (MoE) block that looks at each token’s Hidden state, scores every Expert, and dispatches the token to the top-scoring few. It is trained jointly with the experts, and it is where MoE’s failure modes live: routers can collapse onto a few favorite experts unless Load balancing (MoE) pressure keeps assignments spread out.

Load balancing (MoE)

Load balancing in Mixture of experts (MoE) training counters a natural failure loop: an Expert that gets more tokens learns faster, which makes the Router pick it even more, until a few experts do all the work and the rest waste memory. An auxiliary loss (or a loss-free bias tweak in some recent designs) penalizes uneven assignment, keeping every expert trained and every device in a distributed system busy.

Shazeer et al. 2017, arXiv:1701.06538

Sparse activation

Sparse activation means only a fraction of the model’s Parameters participate in processing any given token — in a Mixture of experts (MoE) model, just the chosen Experts plus the shared layers. This is why MoE models quote two sizes: total parameters (what you store) and active parameters (what each token pays for in compute). Contrast with a Dense model, where every token exercises every weight.

Dense model

A dense model is an ordinary transformer in which every token passes through every Parameter — one Feed-forward network (FFN) per block, no routing. The term exists mainly as the contrast case for Mixture of experts (MoE): dense models are simpler to train and serve and use their memory fully, while MoEs buy more capacity per unit of compute via Sparse activation.

Causal masking

Causal masking zeroes out (by setting to negative infinity before the Softmax) every attention score from a token to positions after it, so information only flows backward-to-forward. This is what lets one training pass supervise every position at once — each token predicts its successor using only its past — and it is why generation must proceed token by token. Encoder models like BERT drop the mask and attend bidirectionally.

Vaswani et al. 2017, arXiv:1706.03762

Attention scores

Attention scores are the raw match values between one token’s query and every other token’s key — dot products, scaled down by the square root of Head dimension to keep the Softmax from saturating. After the softmax they become attention weights: a probability distribution over positions saying how much of each token’s value to mix in. Heat-map visualizations of attention are pictures of these weights.

Vaswani et al. 2017, arXiv:1706.03762

Induction head

An induction head is an Attention head that implements a learned copy rule: find where the current pattern appeared earlier in the context, look at what followed it, and predict that. Interpretability researchers found these heads forming abruptly early in training — across model sizes — and argued they are a core mechanism behind in-context learning, the ability to pick up a pattern from the prompt alone. They are the best-known example of a circuit with a legible job inside a transformer.

Olsson et al. 2022, arXiv:2209.11895

Unembedding

The unembedding (or output projection / LM head) maps a token’s final Hidden state to one score per Vocabulary entry — the Logits. It is the mirror image of the Embedding matrix, and many models literally reuse the same matrix transposed (weight tying) to save parameters. Everything the model “wants to say” has to pass through this single linear map.

Hidden state

A hidden state is the vector a token carries at a given depth in the network — its representation after some number of blocks have read and written to it. Early hidden states stay close to the token’s Embedding; deeper ones blend in context until the final one is handed to the Unembedding to score next tokens. In a transformer, the sequence of hidden states across layers is the Residual stream.

d_model (model dimension)

d_model is the transformer’s width — the length of each token’s Hidden state vector everywhere along the Residual stream. Attention projections, Feed-forward network (FFN) layers, and the Embedding matrix are all sized relative to it, so width (together with depth) largely determines parameter count. Scaling a model up mostly means growing d_model, the number of layers, or both.

Head dimension

The head dimension is the size of one Attention head’s query, key, and value vectors. Conventionally the heads partition the model width, so head_dim is roughly d_model (model dimension) divided by the number of heads — more heads means narrower ones at fixed width. It also appears in the scaling of Attention scores, which are divided by its square root to keep the softmax well-behaved.

Embedding matrix

The embedding matrix is a learned table with one row of width d_model (model dimension) per Vocabulary entry. A token id enters the model by looking up its row — that vector is the token’s starting Hidden state. It is one of the largest single tensors in the model, which is why vocabulary size matters, and it is often tied (transposed) with the Unembedding at the output.

Attention sink

An attention sink is a position — typically the very first tokens of the sequence — that heads pour attention onto regardless of content. Because Softmax weights must sum to one, a head that finds nothing relevant still has to put its weight somewhere, and early tokens become the default dumping ground. The effect is practically important: streaming-inference schemes keep the initial tokens in the KV cache because evicting the sink destabilizes the model.

Xiao et al. 2023, arXiv:2309.17453

Language model

A language model assigns probabilities to sequences of text: given what came before, how likely is each possible continuation? The idea long predates deep learning — statistical N-gram model models powered speech recognition and translation for decades before neural language models, and eventually the Transformer, took over. An Large language model (LLM) is simply a language model made very large.

Bengio et al. 2003, "A Neural Probabilistic Language Model"

N-gram model

An n-gram model predicts the next word from only the previous n−1 words, using frequencies counted in a corpus — a trigram model sees just two words of history. Cheap and interpretable, but it cannot generalize to word combinations it never counted, and its fixed window means no long-range context. Those failure modes are exactly what neural Language models and, later, the Transformer were built to overcome.

Shannon 1948, "A Mathematical Theory of Communication"

RNN (Recurrent Neural Network)

A recurrent neural network processes a sequence one Token at a time, compressing everything seen so far into a fixed-size hidden state. That sequential design made training hard to parallelize and long-range dependencies hard to keep — the LSTM (Hochreiter & Schmidhuber, 1997) eased the memory problem but not the sequential bottleneck. The Transformer replaced recurrence with Attention, processing all tokens in parallel.

Hochreiter & Schmidhuber 1997 (LSTM)

Seq2seq

Sequence-to-sequence (seq2seq) is the encoder–decoder pattern: an encoder network reads the input sequence into a representation, and a decoder generates the output one Token at a time. Built originally from RNNs for machine translation, its fixed-size bottleneck between encoder and decoder motivated the invention of Attention — and the original Transformer was itself a seq2seq model.

Sutskever et al. 2014, arXiv:1409.3215

Cross-entropy

Cross-entropy measures how surprised a model is by the truth: the log of the probability it assigned to the actual next Token, negated. It is the standard Loss for Next-token prediction — zero only if the model put all its probability on the right token, large when the truth was rated unlikely. The concept comes from Shannon’s information theory, where it measures the cost of encoding one distribution using another.

Shannon 1948, "A Mathematical Theory of Communication"

Perplexity

Perplexity is Cross-entropy exponentiated back into an intuitive unit: roughly, how many equally likely options the model is choosing between at each step. A model that always knew the next Token would score 1; pure guessing over the whole Vocabulary scores its size. It is the classic intrinsic measure of language-model quality, though lower perplexity does not guarantee better behavior on real tasks.

Jelinek et al. 1977

Loss

The loss is the number a training run minimizes: a differentiable score of how wrong the model’s predictions are, averaged over batches of data. For LLMs it is the Cross-entropy of Next-token prediction. Gradient descent nudges every Parameter in the direction that lowers it, and the loss curve over a run is the primary health signal engineers watch.

Emergence

Emergence is the observation that some abilities — multi-step arithmetic, instruction following, translation — show up in larger models while smaller ones trained identically show little sign of them. Wei et al. (2022) catalogued such jumps, though later work argues some are artifacts of how the Benchmark is scored rather than sharp changes in the model. Either way, capability at scale is hard to predict from small-model behavior.

Wei et al. 2022, arXiv:2206.07682

Hallucination

A hallucination is model output that is confident and fluent but false — an invented citation, API, or fact. It follows directly from the objective: Next-token prediction rewards plausible continuations, and Sampling will produce one whether or not the model’s Weights encode the underlying fact. Mitigations — retrieval, grounding, citing sources, calibrated refusals — reduce the rate, but no current technique eliminates it.

Knowledge cutoff

The knowledge cutoff is the date at which a model’s training data ends — anything published after it is simply absent from the Weights. Asked about newer events, the model will either say it doesn’t know or produce a Hallucination. The standard workaround is putting fresh information into the Context window at request time, via retrieval or search.

Base model

A base model is what pretraining produces: a pure Next-token prediction engine. Given “What is the capital of France?” it may continue with more exam questions rather than an answer, because continuing text is all it was trained to do. Post-training — instruction tuning and preference optimization — turns a base model into the Instruction-tuned model most people actually use.

Instruction-tuned model

An instruction-tuned model is a Base model fine-tuned on demonstrations of instruction following — request in, helpful answer out — usually followed by preference-based training that shapes tone and safety. This is what chat assistants and APIs serve. The underlying machinery is unchanged: it still does Next-token prediction, just with the distribution steered toward assistant-like continuations.

Checkpoint

A checkpoint is a snapshot of all model Weights — plus, during training, optimizer state — written to disk at a point in a run. Checkpoints let long jobs survive hardware failures and let teams evaluate progress along the way. They are also what ships: a released model is one chosen checkpoint, and the artifact you download as Open weights is a checkpoint someone judged good enough.

Open weights

An open-weights model publishes its trained Weights for download, so anyone with the hardware can run, inspect, or fine-tune it. That is weaker than open source: training data, code, and recipes usually stay private, and licenses may restrict use. The trade-off against Closed weights models is control, privacy, and cost transparency versus the polish and scale of hosted frontier models.

Closed weights

A closed-weights model is served only behind an API; the Weights never leave the provider. You get frontier capability with no infrastructure to run, but no ability to self-host, inspect, or pin the exact artifact — the provider can update or retire the model underneath you. Most production teams weigh this against Open weights alternatives on cost, privacy, and control.

Benchmark

A benchmark is a fixed set of test items with a scoring rule, letting models be compared on the same footing. Benchmarks are indispensable and flawed in equal measure: test items leak into training data (contamination), scores saturate as models improve, and a high score is not the same as competence on your task. Serious evaluation pairs public benchmarks with private, task-specific ones.

Model card

A model card is the standardized documentation released alongside a model: what it is for, how it was trained and evaluated, and where it fails. Proposed by Mitchell et al. (2019) to make model reporting a norm, cards are now the first thing to read when adopting a model — especially the intended-use and limitations sections, and which Benchmark results are actually reported.

Mitchell et al. 2019, "Model Cards for Model Reporting"

Deduplication

Deduplication removes exact and near-duplicate text from a pretraining corpus before training. Web-scale data is riddled with repeats — mirrored pages, boilerplate, syndicated articles — and training on them wastes compute while making verbatim memorization far more likely. Lee et al. (2021) showed that deduplicated data reaches comparable quality in fewer steps and sharply cuts memorized output. A duplicated passage is, in effect, trained on for extra Epochs regardless of what the rest of the Data mixture does.

Lee et al. 2021, arXiv:2107.06499

Data mixture

The data mixture is the recipe of a pretraining run: which sources are included — web crawl, code, books, academic text — and how heavily each is sampled. The mixture shapes what the Base model is good at; upweighting code, for instance, is a deliberate capability decision, not an accident of the crawl. Because a Token budget is finite, mixture weights are among the most consequential and most closely guarded choices in a modern run, typically tuned through small-scale ablations before the real training starts.

Epoch

An epoch is one full pass of training over the dataset. Classic machine learning loops over a small dataset for many epochs, but LLM pretraining is closer to the opposite regime: corpora are so large that a run often sees most Tokens only once, and repeating data brings diminishing returns compared to finding fresh text. This is why Deduplication matters — hidden duplicates silently multiply the epoch count for the repeated text — and why the Data mixture may still deliberately repeat scarce, high-value sources.

Masked language modeling (MLM)

Masked language modeling hides a random subset of input Tokens and trains the model to reconstruct them, letting it attend to context on both sides of each blank. It is the objective behind BERT-style encoders, which excel at understanding tasks but do not generate text left-to-right. Modern LLMs instead use Next-token prediction with Causal masking, which turns every position in every document into a training signal and makes generation native rather than bolted on.

Teacher forcing

Teacher forcing means that during training, the model predicts each next Token conditioned on the true preceding text, not on whatever it would have generated itself. Combined with Causal masking, this lets one forward pass score every position of a sequence in parallel — the trick that makes Next-token prediction trainable at scale. The mismatch it creates is called exposure bias: at inference the model conditions on its own outputs, a distribution it never saw during training.

Scaling laws

Scaling laws are the empirical finding that language-model Loss falls as a smooth power law as you grow Parameter count, dataset size, or training compute. Kaplan et al. (2020) established the pattern, which lets teams forecast the loss of a huge run from a family of cheap small ones — turning “how good will it be?” into an extrapolation rather than a gamble. The follow-up question of how to split a fixed budget between size and data is answered by Compute-optimal analysis.

Kaplan et al. 2020, arXiv:2001.08361

Compute-optimal

A compute-optimal run spends a fixed compute budget on the best trade-off between Parameter count and training Tokens. Hoffmann et al. (2022) found the two should scale roughly in proportion — and proved it with Chinchilla, a smaller model trained on far more data that beat much larger peers at the same budget. Production models today are often trained well past the compute-optimal point, because a smaller model that cost more to train is cheaper to serve for the rest of its life. The result reframed Scaling laws: scale data, not just the model.

Hoffmann et al. 2022, arXiv:2203.15556

FLOPs (FLoating-point OPerations)

FLOPs count floating-point operations — the raw currency of compute in which training budgets, Scaling laws, and even regulation thresholds are quoted. For transformer training a standard estimate is that each Token costs about 6 FLOPs per Parameter (forward plus backward pass), so total compute ≈ 6 × parameters × tokens. Note the pitfall: FLOPs is an amount of work, while FLOP/s is a rate — what an accelerator can do per second — and Model FLOPs utilization (MFU) measures how much of that rate a run actually uses.

Model FLOPs utilization (MFU) (Model FLOPs Utilization)

Model FLOPs utilization is the efficiency score of a training run: the FLOPs the model arithmetic actually needs per second, divided by the hardware’s theoretical peak. Everything that is not model math — communication between devices, pipeline bubbles, memory stalls, recomputation — drags it below 1, and large distributed runs land well under peak. Raising MFU is the day job of pretraining infrastructure teams, because at cluster scale a few points of utilization is worth millions in compute.

Data parallelism

Data parallelism runs a full copy of the model on every device, gives each copy a different slice of the batch, and averages the gradients (an all-reduce) so the Weights stay identical everywhere. It is the simplest way to scale training throughput, but each device must hold the entire model plus optimizer state — which is exactly the redundancy that ZeRO and Fully sharded data parallel (FSDP) shard away. When the model itself no longer fits on one device, it is combined with Tensor parallelism and Pipeline parallelism.

Tensor parallelism

Tensor parallelism splits individual weight matrices across devices, so a single layer’s matrix multiply is computed jointly by all of them — parallelism within a layer rather than across the batch or the depth of the model. Popularized by Megatron-LM, it lets Transformer layers too large for one accelerator run at all, but the devices must exchange activations inside every layer, so it is usually confined to the fast interconnect within one node. It slots into the standard 3D recipe alongside Data parallelism and Pipeline parallelism.

Shoeybi et al. 2019, arXiv:1909.08053

Pipeline parallelism

Pipeline parallelism cuts the model by depth: consecutive groups of layers become stages living on different devices, and activations flow stage to stage like an assembly line. Introduced at scale by GPipe, it splits each batch into micro-batches streamed through the pipeline so stages work concurrently — yet some idle time (the “pipeline bubble”) at the start and end of each step is unavoidable. Because stages only communicate at their boundaries, it tolerates slower links than Tensor parallelism and is combined with it and Data parallelism in large runs.

Huang et al. 2018, arXiv:1811.06965

ZeRO (Zero Redundancy Optimizer)

ZeRO keeps the training math of Data parallelism but removes its memory redundancy: instead of every device holding a full replica, optimizer state, gradients, and finally the Weights themselves are partitioned across the group and gathered only when needed. Its three stages shard progressively more — stage 1 the optimizer state, stage 2 gradients too, stage 3 the parameters as well. Introduced with DeepSpeed, the idea became the template for Fully sharded data parallel (FSDP) and made models far larger than one device’s memory trainable with familiar data-parallel code.

Rajbhandari et al. 2019, arXiv:1910.02054

Fully sharded data parallel (FSDP) (Fully Sharded Data Parallel)

Fully sharded data parallel is PyTorch’s native realization of the ZeRO idea: Weights, gradients, and optimizer state are sharded across the data-parallel group, and each layer’s full parameters exist only momentarily — gathered just before their forward or backward computation, then freed. That trades extra communication for a dramatic drop in per-device memory, making it the default way to fine-tune or pretrain large models on ordinary Data parallelism-style clusters without a separate framework.

Mixed precision

Mixed-precision training does the bulk of its math in 16-bit floats — roughly halving memory and unlocking the fast matrix hardware of modern accelerators — while keeping an FP32 master copy of the Weights and doing sensitive operations at full precision. The original FP16 recipe needed loss scaling to stop tiny gradients from vanishing to zero; the wider exponent range of BF16 (bfloat16) largely removed that chore, and 16-bit compute is now simply how pretraining is done.

Micikevicius et al. 2017, arXiv:1710.03740

BF16 (bfloat16) (Brain Floating Point 16)

BF16 is a 16-bit floating-point format from Google Brain that keeps FP32’s 8 exponent bits and pays for it with a short 7-bit mantissa — the opposite trade from FP16, which has more precision but a narrow exponent range. The wide range means values almost never overflow or underflow during training, so Mixed precision with BF16 usually needs no loss scaling. That robustness — fewer numerics-induced Loss spikes for the same memory cost — made it the default 16-bit format for pretraining Large language model (LLM)s.

Loss spike

A loss spike is a sudden upward jump in the training Loss of an otherwise healthy run — the curve that was grinding downward abruptly leaps. Causes are debated and often entangled: a pathological stretch of data, numerical edge cases, or optimizer state gone brittle, with large models and high Learning rates more susceptible. The working playbook is pragmatic: monitor for spikes, and when one refuses to recover, rewind to an earlier Checkpoint and skip past the offending batches. Loss spikes are a core reason pretraining runs are babysat around the clock.

Learning rate

The learning rate scales how far every Parameter moves along its gradient at each step: too high and the Loss diverges or spikes, too low and the run crawls and settles worse. Pretraining never uses a constant value — a warmup ramps it up from near zero so early noisy gradients cannot wreck the initialization, then a long decay (cosine is the classic shape) anneals it toward the end of the run. Tuning the peak value and schedule is among the highest-stakes decisions made before a run, since it cannot be cheaply redone after.

Supervised fine-tuning (SFT) (Supervised Fine-Tuning)

Supervised fine-tuning (SFT) trains a Base model on curated prompt–response pairs, reusing the Next-token prediction objective but typically computing the Loss only on the response tokens the model should imitate. It is the first post-training stage: after SFT the model answers requests in the assistant format rather than just continuing the text. Because SFT can only imitate its demonstrations, preference methods like Reinforcement learning from human feedback (RLHF) are what push quality beyond the demonstrators.

Instruction tuning

Instruction tuning fine-tunes a Base model on datasets of instructions paired with good responses, turning “continue this text” into “do what this text asks”. It is what separates an Instruction-tuned model from its base: the same architecture, reshaped to follow requests — including kinds of requests the tuning data never covered. Ouyang et al. (2022) combined it with Reinforcement learning from human feedback (RLHF) in the InstructGPT pipeline that set the template for modern assistants.

Ouyang et al. 2022, arXiv:2203.02155

Chat template

A chat template is the exact serialization a chat model was trained on: special Tokens and role markers that flatten system, user, and assistant turns into one sequence. The model never sees “a conversation” — only these tokens — so running a model with the wrong template silently degrades it. The template is also how a System prompt is spliced in and how the model knows where its own turn begins and ends.

System prompt

A system prompt is instruction text placed at the start of the context in a dedicated role slot of the Chat template, which post-training has taught the model to treat with elevated authority. It is the cheapest steering layer: deployers set persona, rules, tone, and tool definitions there without touching the weights. That authority is trained rather than enforced, so adversarial user input can sometimes override it — the root of prompt injection, and a reason Refusal behavior cannot live in the prompt alone.

Reinforcement learning from human feedback (RLHF) (Reinforcement Learning from Human Feedback)

Reinforcement learning from human feedback (RLHF) optimizes a model against a Reward model trained to predict human preferences between candidate responses. The policy generates, the reward model scores, and an RL algorithm (classically PPO) updates the weights, with a KL penalty keeping the policy close to its Supervised fine-tuning (SFT) starting point. Ouyang et al. (2022) used this pipeline to build InstructGPT, showing that raters preferred a far smaller fine-tuned model over a much larger Base model.

Ouyang et al. 2022, arXiv:2203.02155

Reward model

A reward model takes a prompt and a candidate response and outputs a scalar score for how much a rater would prefer it. It is trained on Preference data — comparisons where one response was marked better — and then stands in for the rater during Reinforcement learning from human feedback (RLHF), scoring millions of samples no human could review. Because it is only a proxy for real preferences, optimizing against it too hard invites Reward hacking.

Christiano et al. 2017, arXiv:1706.03741

Preference data

Preference data consists of comparisons: a prompt, two or more candidate responses, and a label for which one a rater preferred. Christiano et al. (2017) built RL around such comparisons because people answer “which is better?” far more consistently than they score quality on an absolute scale. Preference pairs are the raw material for both the Reward model in Reinforcement learning from human feedback (RLHF) and the direct loss in Direct preference optimization (DPO).

Christiano et al. 2017, arXiv:1706.03741

KL penalty

The KL penalty adds a cost to the Reinforcement learning from human feedback (RLHF) objective proportional to the Kullback–Leibler divergence between the policy being trained and a frozen reference model, usually the Supervised fine-tuning (SFT) checkpoint. Without it, the policy chases Reward model scores into degenerate, off-distribution text; with it, reward can only be earned in ways that stay close to fluent language. Direct preference optimization (DPO) bakes the same KL-constrained trade-off directly into its loss.

Reward hacking

Reward hacking is a policy exploiting flaws in its reward signal: scoring ever higher on the Reward model while getting worse by the standard the reward was meant to proxy. Classic LLM symptoms are sycophancy, padded length, and confident Hallucination, because raters — and the models trained on their labels — systematically over-reward agreeable, long, assertive answers. The KL penalty, early stopping, and fresh preference data are the standard defenses.

Direct preference optimization (DPO) (Direct Preference Optimization)

Direct preference optimization (DPO) fine-tunes directly on Preference data with a classification-style Loss that raises the likelihood of chosen responses over rejected ones, relative to a frozen reference model. Rafailov et al. (2023) showed this optimizes the same KL-constrained objective as Reinforcement learning from human feedback (RLHF) while skipping the separate Reward model and the unstable RL sampling loop. That simplicity made it the default preference method for open-weights models and spawned a family of variants.

Rafailov et al. 2023, arXiv:2305.18290

Reinforcement learning from AI feedback (RLAIF) (Reinforcement Learning from AI Feedback)

Reinforcement learning from AI feedback (RLAIF) follows the Reinforcement learning from human feedback (RLHF) recipe but replaces human preference labels with judgments from an LLM. A capable model compares candidate responses — often against written principles, as in Constitutional AI — and its verdicts become the Preference data that trains the Reward model. Because AI labels are cheap and fast, RLAIF scales feedback far beyond human annotation budgets, at the cost of inheriting the judge’s blind spots.

Bai et al. 2022, arXiv:2212.08073

Constitutional AI

Constitutional AI trains harmlessness from a written list of principles instead of per-example human harm labels. In Bai et al. (2022), the model first critiques and revises its own responses against the constitution — Supervised fine-tuning (SFT) on the revisions — and then an AI judge compares response pairs against the principles, producing the Preference data for an Reinforcement learning from AI feedback (RLAIF) phase. The constitution makes the training target explicit and auditable: you can read the values rather than infer them from behavior.

Bai et al. 2022, arXiv:2212.08073

Reasoning model

A reasoning model is post-trained — typically with reinforcement learning on problems whose answers can be checked, like math and code — to produce a long Chain-of-thought (CoT) before its final answer. That shifts scaling to inference: accuracy on hard problems climbs with the Token budget the model is allowed to think in, not just with model size. Models like OpenAI’s o1 and DeepSeek-R1 established the recipe, rewarding checkable final answers rather than imitating human-written reasoning.

Chain-of-thought (CoT)

Chain-of-thought (CoT) is intermediate reasoning written out in Tokens before the final answer. Wei et al. (2022) showed that prompting large models with worked examples of step-by-step reasoning dramatically improves arithmetic and multi-step Benchmark performance — an ability that emerges with scale. What began as a prompting trick became a training target: a Reasoning model is optimized to generate long chains of thought on its own. The written steps also give useful, if imperfect, visibility into how an answer was reached.

Wei et al. 2022, arXiv:2201.11903

Alignment

Alignment is making a model reliably do what its principals intend — commonly summarized as helpful, honest, and harmless — rather than whatever its raw training objective happens to reward. In practice it is the umbrella over post-training: Reinforcement learning from human feedback (RLHF), Constitutional AI, and Refusal training all target it. The hard part is that every objective is a proxy, and gaps between proxy and intent surface as sycophancy, Reward hacking, and confidently wrong answers.

Alignment tax

The alignment tax is whatever capability a model loses as a side effect of Alignment training — Benchmark regressions after preference tuning, or over-cautious Refusal of benign requests. The InstructGPT work measured such regressions and showed that mixing pretraining data into the RL phase could shrink them. How much tax to accept — and how much of it is real capability loss versus changed behavior — is a live trade-off in every production deployment.

Refusal

A refusal is the model declining to fulfill a request — the front-line behavior that safety post-training installs for harmful queries. It is learned, not architectural: Supervised fine-tuning (SFT) examples and preference signals teach both when to refuse and how, which is why refusals can be inconsistent and why jailbreaks that shift the apparent context can route around them. Placing the boundary is a core Alignment trade-off — too loose enables misuse, too strict over-refuses benign requests, a visible Alignment tax.

Prefill

Prefill is the phase of inference that ingests the prompt: every input Token is processed in a single parallel pass, computing the keys and values that populate the KV cache along with the Logits for the first output token. Because all positions are computed at once, prefill is compute-bound, and its cost grows with prompt length. Its duration is what a user feels as TTFT — the wait before anything appears on screen.

Decode

Decode is the generation phase that follows Prefill: the model emits one Token per forward pass, reading every earlier position’s keys and values from the KV cache and appending its own. Each step depends on the previous one, so decode is inherently sequential — and memory-bandwidth-bound rather than compute-bound, since most of each step is spent streaming Weights and cache rather than multiplying. Its per-step pace is what TPOT measures.

TTFT (Time To First Token)

Time To First Token is the latency from request arrival to the first generated Token — the pause before a streaming response starts moving. It is dominated by Prefill, so it grows with prompt length, plus whatever time the request spends queued inside the Serving engine. Interactive products watch TTFT closely because a response that starts quickly feels fast even when it streams slowly; Prefix caching is one of the main levers for cutting it.

TPOT (Time Per Output Token)

Time Per Output Token is the average time each additional Token takes during Decode — the cadence of a streaming response once TTFT has passed. It reflects how memory-bandwidth-bound each decode step is and how heavily the batch is loaded: packing more requests together raises Throughput but can stretch every user’s TPOT. Total response latency is roughly TTFT plus TPOT times the number of output tokens.

Throughput

Throughput measures the aggregate output of a serving system — typically Tokens generated per second summed over every concurrent request, sometimes requests completed per second. It trades off against per-request latency: batching more requests keeps the accelerator busy and raises throughput, but each user’s TPOT grows. Techniques like Continuous batching and PagedAttention exist largely to push that frontier outward.

Goodput

Goodput refines Throughput by counting only the work that meets its service-level objectives — for example, requests whose TTFT and TPOT stayed under target. A system can post an impressive raw token rate while violating latency targets for a large share of users; goodput exposes that gap. It is the more honest capacity-planning metric for interactive serving, where a token delivered too late is close to worthless.

Temperature

Temperature divides the Logits by a constant before the Softmax, reshaping the next-token distribution ahead of Sampling. Values below 1 sharpen it — probability mass concentrates on the most likely tokens — while values above 1 flatten it, admitting more surprising choices; as temperature approaches 0, sampling converges to Greedy decoding. It changes how the distribution is drawn from, not what the model has learned.

Top-k sampling

Top-k sampling truncates the next-token distribution to the k most probable Tokens, renormalizes, and samples among them, so the long tail of unlikely tokens can never be drawn. Introduced by Fan et al. (2018) for story generation, it curbs the derailments that unrestricted Sampling produces when a rare token gets picked. Its weakness is that k is fixed: the same cutoff applies whether the model is confident (making k too generous) or uncertain (making k too strict) — the problem Top-p (nucleus) sampling was designed to fix.

Fan et al. 2018, arXiv:1805.04833

Top-p (nucleus) sampling

Top-p sampling (nucleus sampling) keeps the smallest set of Tokens whose cumulative probability reaches p and samples within that nucleus. Unlike Top-k sampling, the candidate count adapts: a confident distribution may leave only a token or two, an uncertain one may leave hundreds. Holtzman et al. (2020) proposed it after showing that likelihood-maximizing decoding turns repetitive while untruncated Sampling turns incoherent — the nucleus keeps the plausible middle. In practice it is combined with Temperature and exposed as top_p in most APIs.

Holtzman et al. 2020, arXiv:1904.09751

Greedy decoding

Greedy decoding takes the argmax of the next-token distribution at every step — no randomness, so the same prompt yields the same output (up to hardware nondeterminism). Each step is locally optimal, but the sequence as a whole is not, and greedy generation famously drifts into repetition loops on open-ended text. It suits tasks with essentially one right answer; Beam search generalizes it by tracking several candidates, and Sampling with Temperature replaces it when diversity matters.

Beam search

Beam search maintains the b best partial sequences (the beam) at every step, extending each and keeping the top scorers, in search of a higher-probability completion than Greedy decoding’s single path can find. It was the standard in machine translation, where outputs are short and tightly constrained by the input. For open-ended LLM generation it has fallen out of favor: maximizing sequence probability yields bland, repetitive text — a finding that motivated Top-p (nucleus) sampling Sampling — and carrying b hypotheses multiplies compute and KV cache use.

PagedAttention

PagedAttention stores the KV cache in fixed-size blocks allocated on demand, with an indirection table mapping each sequence’s logical positions to physical blocks — operating-system paging applied to accelerator memory. Reserving contiguous memory for every request’s maximum possible length wastes much of it on fragmentation; paging lets far more concurrent requests fit, and blocks can be shared between sequences for Prefix caching. Introduced by Kwon et al. (2023), it is the core idea of vLLM and is now standard across Serving engines.

Kwon et al. 2023, arXiv:2309.06180

Prefix caching

Prefix caching keeps the KV cache entries computed for a prompt prefix and reuses them whenever another request begins with exactly the same Tokens — a long system prompt or shared document then goes through Prefill only once. Because each cached position depends only on the tokens before it, reuse is sound for identical prefixes but stops at the first token that differs. For prompt-heavy workloads it can cut TTFT and cost substantially, which is why providers expose it as “prompt caching”.

Quantization

Quantization represents a model’s Weights — and sometimes activations or the KV cache — in lower-precision formats such as 8-bit or 4-bit integers instead of 16-bit floats. Because Decode is memory-bandwidth-bound, cutting the bytes moved per step both shrinks the hardware needed to hold a model and speeds up generation. Post-training methods like GPTQ and AWQ quantize a finished model without retraining, accepting a small, method-dependent accuracy cost. It differs from Mixed precision, which lowers precision during training rather than for deployment.

GPTQ

GPTQ is a one-shot post-training Quantization method from Frantar et al. (2022) that compresses a model’s Weights to 3–4 bits with no retraining. It works layer by layer, using approximate second-order (Hessian) information from a small calibration set to update the remaining weights and compensate for each rounding error. The name comes from the paper that introduced it — quantization aimed at GPT-family models — and it became one of the standard formats for running large open-weight models on modest hardware, alongside AWQ.

Frantar et al. 2022, arXiv:2210.17323

AWQ (Activation-aware Weight Quantization)

Activation-aware Weight Quantization (Lin et al. 2023) starts from the observation that a small fraction of Weights — those multiplied by large activations — matter disproportionately for model quality. Rather than keeping them in higher precision, AWQ scales those channels up (and their inputs down) before Quantization, preserving accuracy at low bit-widths without retraining or backpropagation. Because it needs no gradient-based reconstruction, it is simple and fast to apply, and it sits alongside GPTQ as a standard recipe for low-bit open-weight models.

Lin et al. 2023, arXiv:2306.00978

Speculative decoding

Speculative decoding attacks the sequential bottleneck of Decode: a fast Draft model proposes a short run of Tokens, and the target model scores them all in a single parallel forward pass, accepting the prefix it agrees with. A rejection-Sampling rule (Leviathan et al. 2022) makes the technique exact — the output distribution is provably identical to decoding with the target model alone. The speedup depends on how often drafts are accepted, so it shines when the draft is cheap and usually right; the price is extra compute, never quality.

Leviathan et al. 2022, arXiv:2211.17192

Draft model

A draft model is the small companion model in Speculative decoding: cheap enough to run several Decode steps in the time the target model takes for one, yet similar enough that its guesses are usually accepted. It can be a smaller model from the same family, a distilled copy of the target, or extra prediction heads attached to the target itself. The better its agreement with the target, the more Tokens are accepted per verification pass — and the larger the end-to-end speedup.

Leviathan et al. 2022, arXiv:2211.17192

Continuous batching

Continuous batching schedules work per iteration rather than per request: after each Decode step, completed sequences exit the batch and queued requests are admitted immediately. Static batching wastes the accelerator whenever short responses finish early and their slots sit idle while the longest one drags on; continuous batching keeps every slot doing useful work, which is why it lifts Throughput so sharply. It is the scheduling backbone of modern Serving engines and pairs naturally with PagedAttention, which makes the ever-changing batch’s KV cache memory easy to allocate and free.

Serving engine

A serving engine is the inference runtime standing between model Weights and an API endpoint: it handles request scheduling, Continuous batching, KV cache management (typically via PagedAttention), optimized attention kernels, and Quantization formats. Engines such as vLLM, TensorRT-LLM, and SGLang differ in kernels and scheduling policy but share this shape. For a given model and GPU, the engine — as much as the hardware — determines the Throughput and latency a deployment achieves.

Sliding-window attention

Sliding-window attention lets each Token attend only over a fixed window of the most recent positions rather than the full Context window, so per-step Attention cost and KV cache size stop growing with sequence length. Information from beyond the window can still propagate, because each layer’s window looks at representations that already summarize an earlier window — the effective receptive field widens with depth. Mistral 7B used it to serve long inputs cheaply, and later designs often interleave windowed and full-attention layers to balance reach against cost. Like Grouped-query attention (GQA), it is an architectural choice made largely for inference economics.

Zero-shot

Zero-shot prompting asks the model to perform a task it was never shown an example of — the instruction in the Context window is all it gets. Instruction-tuned models handle this surprisingly well because Supervised fine-tuning (SFT) taught them to follow directions in general. When zero-shot output is unreliable, adding demonstrations (Few-shot) is usually the cheapest next lever.

Few-shot

Few-shot prompting places a handful of demonstrations — input and desired output pairs — in the Context window before the real query. The model picks up the format, style, and decision boundary from the examples via In-context learning, with no weight updates involved. Demonstrations can also include worked reasoning, which shades into Chain-of-thought (CoT) prompting. Brown et al. (2020) showed the approach works dramatically better as models scale.

Brown et al. 2020, arXiv:2005.14165

In-context learning

In-context learning is the ability of large models to infer a task from demonstrations in the prompt and perform it — with the Weights completely frozen. It emerged with scale rather than being explicitly trained for, and it is the mechanism that makes Few-shot prompting work. The learning is ephemeral: it exists only within that Context window and vanishes with it.

Brown et al. 2020, arXiv:2005.14165

Context engineering

Context engineering treats the Context window as an engineered artifact: choosing what goes in (System prompt, retrieved documents, conversation history, tool results), in what order, and what gets summarized or dropped as the Token budget fills. It generalizes prompt engineering from wording a request to curating the model’s entire working set. Techniques like Retrieval-augmented generation (RAG) are context engineering with a retrieval step attached.

Retrieval-augmented generation (RAG) (Retrieval-Augmented Generation)

Retrieval-augmented generation bolts a search step onto generation: a Retrieval system fetches passages relevant to the query, and they are inserted into the Context window before the model answers. Named by Lewis et al. (2020), RAG lets a frozen model use fresh or private knowledge, and it reduces Hallucination by Grounding answers in retrievable evidence. Its weak point is the retrieval itself: the model can only be as right as the passages it was handed.

Lewis et al. 2020, arXiv:2005.11401

Retrieval

Retrieval is the search stage of Retrieval-augmented generation (RAG): given a query, find the most relevant chunks in a corpus. Lexical retrieval (like BM25) matches keywords; dense retrieval compares Embedding vectors so paraphrases match too. Everything downstream depends on it — a perfect generator cannot recover from retrieval that fetched the wrong passages.

Embedding model

An embedding model encodes a text into a single Embedding vector such that semantically similar texts land near each other. Because each document is encoded independently (a bi-encoder), a whole corpus can be embedded once, indexed, and searched with fast Approximate nearest neighbor (ANN) methods. Sentence-BERT (Reimers & Gurevych 2019) established this recipe, and it remains the workhorse of dense Retrieval.

Reimers & Gurevych 2019, arXiv:1908.10084

Cross-encoder

A cross-encoder feeds the query and a candidate document through the model together, letting attention compare them token by token before emitting a relevance score. That joint reading beats Embedding model similarity on accuracy, but it must run once per query–document pair, so it cannot search millions of documents. The standard compromise is Reranking: a fast retriever fetches candidates, and the cross-encoder reorders the shortlist.

Reimers & Gurevych 2019, arXiv:1908.10084

Chunking

Chunking splits documents into the pieces that get embedded and retrieved, since whole documents are too long to embed faithfully or to fit the Token budget. It is a real trade-off: small chunks give precise matches but strand sentences without their context, while large chunks dilute the Embedding and drag irrelevant text into the prompt. Common tactics include overlapping windows, splitting on document structure, and attaching parent-document context to each chunk.

Vector database

A vector database stores Embedding vectors alongside their source text and metadata, and answers “which stored vectors are closest to this query vector?” using Approximate nearest neighbor (ANN) indexes such as HNSW. Beyond similarity search it typically adds metadata filtering, hybrid keyword-plus-vector queries, and the operational features of a normal database. It is the persistence layer of most Retrieval-augmented generation (RAG) deployments.

Approximate nearest neighbor (ANN) (Approximate Nearest Neighbor)

Approximate nearest neighbor search finds vectors close to a query without exhaustively comparing against every stored Embedding, which would scale linearly with corpus size. Index structures — graphs like HNSW, inverted files, quantized codes — return almost-always-correct neighbors at a tiny fraction of the cost. The recall-versus-speed dial is a first-class tuning knob in every Vector database.

HNSW (Hierarchical Navigable Small World)

HNSW organizes vectors into a hierarchy of graph layers: upper layers are sparse and support long hops, lower layers are dense and local. A query greedily walks each layer toward the nearest neighbors found so far, then descends to refine. Introduced by Malkov & Yashunin (2016), it is the default Approximate nearest neighbor (ANN) index in most Vector database engines, trading memory for excellent speed and recall.

Malkov & Yashunin 2016, arXiv:1603.09320

Hybrid search

Hybrid search runs lexical Retrieval (keyword matching such as BM25) and dense retrieval (via an Embedding model) in parallel and merges the ranked lists, often with reciprocal rank fusion. Each side covers the other’s blind spot: embeddings miss exact part numbers and rare names, while keywords miss paraphrases. Much production Retrieval-augmented generation (RAG) retrieval is hybrid, frequently followed by Reranking.

Reranking

Reranking takes the candidate list from a fast first-stage Retrieval and rescores it with a stronger model — typically a Cross-encoder — before the top results enter the Context window. The two-stage design gets the accuracy of expensive pairwise scoring while only paying for it on a shortlist. It is often the single highest-leverage upgrade to a mediocre RAG pipeline.

Grounding

Grounding means constraining a model’s answer to evidence it was actually given — retrieved passages, tool results, documents — rather than whatever its weights happen to recall. A grounded system can cite where each claim came from, which turns Hallucination from an invisible failure into a checkable one. Retrieval-augmented generation (RAG) is the standard grounding mechanism, but grounding is the goal; retrieval is just one way to get there.

Fine-tuning

Fine-tuning continues training a pretrained model on a smaller, task-specific dataset, actually changing its Weights — most commonly via Supervised fine-tuning (SFT) on curated examples. It excels at teaching behavior: output format, tone, domain vocabulary, tool conventions. For knowledge that changes, Retrieval-augmented generation (RAG) usually beats it, since baked-in facts go stale and are hard to attribute; parameter-efficient methods like LoRA make the mechanics cheap enough to iterate on.

LoRA (Low-Rank Adaptation)

LoRA freezes the pretrained Weights and injects trainable low-rank matrix pairs into selected layers; only these small matrices are updated during Fine-tuning. Because the update has low rank, trainable parameters drop by orders of magnitude, and the learned delta can be merged into the base weights at inference — no added latency. Introduced by Hu et al. (2021), it is the default recipe for adapting open-weight LLMs, with adapters small enough to keep one per task.

Hu et al. 2021, arXiv:2106.09685

QLoRA (Quantized LoRA)

QLoRA combines Quantization and LoRA: the frozen Base model is stored in 4-bit precision (the NF4 data type) while the small LoRA matrices train in higher precision on top. Gradients flow through the quantized weights into the adapters, cutting fine-tuning memory dramatically at accuracy close to ordinary LoRA. Dettmers et al. (2023) introduced it, putting large-model fine-tuning within reach of single-GPU budgets.

Dettmers et al. 2023, arXiv:2305.14314

Adapter

An adapter is a small trainable module attached to a frozen pretrained network so that task learning happens in the added parameters, not the original Weights. Adapters founded the family now called parameter-efficient fine-tuning (PEFT), of which LoRA is the dominant member. Because each adapter is tiny, one Base model can serve many tasks by swapping adapters instead of storing full model copies.

Houlsby et al. 2019, arXiv:1902.00751

Distillation

Distillation transfers capability from a large teacher model into a smaller student by training the student to match the teacher’s outputs. Hinton et al. (2015) framed it as matching the teacher’s softened probability distribution, which carries far more signal than hard labels. In modern LLM practice it usually means generating Synthetic data with a strong teacher and running Supervised fine-tuning (SFT) on the student — the standard route to small models that punch above their size.

Hinton et al. 2015, arXiv:1503.02531

Synthetic data

Synthetic data is training data produced by a model: generated instructions, answers, reasoning traces, or preference pairs used for Supervised fine-tuning (SFT) and Distillation. It scales where human labeling cannot, and it can be filtered or verified programmatically. The risks are inherited bias and Hallucination from the generator, plus degradation when models recursively train on their own unfiltered output — so filtering and verification carry most of the value.

Eval harness

An eval harness is the infrastructure that runs an evaluation end to end: it loads test cases, calls the model, passes each output to a Grader, and aggregates results into scores. Treating it as software — a versioned Golden set, repeatable grading, comparable reports — is what turns “the model seems better” into a measurement. The same harness that compares models also powers Regression testing before any prompt or model change ships.

Golden set

A golden set is a hand-curated collection of test inputs paired with verified, known-good answers — the ground truth an Eval harness scores against. Unlike a public Benchmark, it is built from your own domain and observed failure cases, so it measures what your users actually need. Like any Held-out set, it stops being trustworthy once its examples leak into Fine-tuning data or the prompt-iteration loop.

Contamination

Contamination is the leakage of evaluation data into a model’s training corpus — the model has effectively seen the exam. A contaminated Benchmark inflates scores because the model can recall answers instead of deriving them, which is why results on aging public test sets grow less trustworthy over time. Web-scale pretraining makes some leakage nearly unavoidable, so serious evaluations lean on a private Held-out set or freshly written items.

Held-out set

A held-out set is data deliberately withheld from training and reserved for measurement. Because the model never saw it, performance on it estimates generalization — the thing evaluation exists to measure. The discipline is easy to break in practice: iterating on prompts against the same set, or letting examples slip into Fine-tuning data, quietly converts it into training data — Contamination by another route.

Elo rating

An Elo rating turns pairwise win/loss outcomes into a single skill score — originally for chess players, now for models battling in an Arena. Each head-to-head vote nudges the winner up and the loser down, with upsets against higher-rated opponents worth more. The result is a relative ranking, not an absolute measure: an Elo number only means something next to other models rated on the same pool of comparisons.

Arena

An arena evaluates models by battle: a user sends one prompt to two anonymous models, sees both responses side by side, and votes for the better one. Aggregated over many users, the votes become Elo rating-style leaderboard scores grounded in real human preference on fresh prompts — which also sidesteps Contamination. Chatbot Arena (Chiang et al. 2024) is the canonical example. The trade-off: votes reward what users like — confidence, formatting, length — which is not always what is correct.

Chiang et al. 2024, arXiv:2403.04132

LLM-as-judge

LLM-as-judge uses a strong model to score another model’s outputs, either against a Rubric or by picking the better of two responses. Zheng et al. (2023) showed that strong judges can agree with human preferences at rates comparable to human–human agreement, which made judging the workhorse Grader for open-ended tasks where string matching fails. The catch is Judge bias: judges systematically favor certain positions, lengths, and styles, so serious harnesses measure and correct for those tendencies.

Zheng et al. 2023, arXiv:2306.05685

Judge bias

Judge bias is the set of systematic errors an LLM-as-judge makes: position bias (favoring whichever answer appears first), verbosity bias (favoring longer answers), and self-enhancement bias (favoring outputs in the judge’s own style). Zheng et al. (2023) documented these along with the standard mitigations — swapping answer order and averaging, requiring reasoning before a verdict, anchoring the judge to a Rubric and reference answer. Left uncorrected, judge bias means an eval ranks models by how well they flatter the judge, not how well they perform.

Zheng et al. 2023, arXiv:2306.05685

Rubric

A rubric spells out what a good answer looks like: the criteria to check, the scale to score on, and often worked examples of each grade. It converts a vague instruction like “rate this response” into a repeatable procedure — which matters doubly for an LLM-as-judge, where an underspecified prompt lets Judge bias fill the gaps. Good rubrics are task-specific and versioned alongside the Eval harness, because changing the rubric changes the metric.

Grader

A grader is whatever scores model output inside an Eval harness: exact string match, unit tests for code (as in HumanEval), a Rubric-guided LLM-as-judge, or a human reviewer. Choosing one is the central design decision of an eval — cheap deterministic graders only fit tasks with a single right answer, while open-ended tasks force you onto judges whose reliability must itself be measured. The common failure is silently trusting the grader; graders need periodic spot-checks against human labels.

Regression testing

Regression testing for LLM systems re-runs a fixed suite — typically a Golden set scored by an Eval harness — whenever anything changes: the prompt, the model version, the retrieval index, a dependency. Because model behavior is not modular, an edit that fixes one case can quietly break ten others, and only a broad re-run catches it. It is the habit that most separates production LLM engineering from demos.

Drift

Drift is degradation that arrives without a deploy: user traffic shifts away from what the Golden set covers, the world changes so once-correct answers go stale, or a provider updates the model behind an API. Because nothing in your repo changed, Regression testing against a fixed set can miss it — detection requires continuously sampling and grading live traffic. Drift is the argument for treating evaluation as an ongoing operation rather than a pre-launch gate.

MMLU (Massive Multitask Language Understanding)

MMLU (Hendrycks et al. 2020) tests knowledge with multiple-choice questions across 57 subjects, from elementary math to professional law. Its breadth made it the field’s default headline Benchmark for years — the one number press releases quote. Age is its weakness: public since 2020, it is a prime target for Contamination, and as top models cluster near its ceiling, successors and private Held-out sets carry more signal.

Hendrycks et al. 2020, arXiv:2009.03300

HumanEval

HumanEval (Chen et al. 2021, released alongside Codex) is a set of hand-written Python programming problems, each with hidden unit tests. Grading is execution, not text similarity: generated code either passes the tests or it does not, reported via pass@k. The name means the problems were written by humans — to keep them out of training corpora and dodge Contamination — not that humans do the grading.

Chen et al. 2021, arXiv:2107.03374

HELM (Holistic Evaluation of Language Models)

HELM (Liang et al. 2022, Stanford CRFM) argues that a single Benchmark number hides too much, and instead evaluates models across a grid of scenarios and metrics — accuracy alongside calibration, robustness, fairness, bias, toxicity, and efficiency. Every model runs under the same conditions with results published transparently, making it as much an evaluation methodology as a leaderboard. Its lasting influence is the norm — echoed in every Model card — that model quality is a profile, not a score.

Liang et al. 2022, arXiv:2211.09110

pass@k

pass@k measures code generation by Sampling multiple candidate solutions per problem and asking whether at least one of k passes the unit tests. Chen et al. (2021) introduced the unbiased estimator used today: generate n ≥ k samples, count the passes, and compute the expectation rather than literally drawing k. Because it rewards diverse attempts, pass@k interacts with Temperature — higher k favors more exploratory sampling — so vendor numbers are only comparable at the same k and settings.

Chen et al. 2021, arXiv:2107.03374

Prompt injection

Prompt injection exploits the LLM’s core weakness: instructions and data arrive in one undifferentiated stream of Tokens, so text an attacker controls can override what the developer intended. A direct injection arrives through the user’s own message; Indirect prompt injection hides the payload in content the model is asked to process. Because there is no hardware-enforced privilege boundary inside the prompt, defenses like the Instruction hierarchy and System prompt hardening reduce the risk rather than eliminate it.

Indirect prompt injection

Indirect prompt injection is the variant of Prompt injection where the payload rides in material the model will later read: a webpage it browses, an email it summarizes, a document surfaced by Retrieval-augmented generation (RAG). The user never sees the attack; it enters the Context window as apparently innocent data and the model treats it as instructions. Greshake et al. (2023) showed this turns any LLM that reads untrusted content into a potential confused deputy — which is why tool-using agents make the problem urgent.

Greshake et al. 2023, arXiv:2302.12173

Jailbreak

A jailbreak manipulates a model into setting aside its safety training — role-play framing, hypothetical wrappers, encoding tricks, or many-turn escalation. It differs from Prompt injection in its target: a jailbreak attacks the model’s own Alignment, while injection attacks the application built around the model. Because Reinforcement learning from human feedback (RLHF) shapes behavior statistically rather than by rule, Refusal generalizes imperfectly and new jailbreaks keep being found.

Adversarial suffix

An adversarial suffix is a sequence of Tokens found by automated optimization — not human creativity — that, appended to a harmful request, defeats a model’s Refusal training. Zou et al. (2023) showed such suffixes can be built by gradient-guided search on open-weight models and often transfer to closed models they were never optimized against. The result reframed Jailbreaking as an optimization problem: defenses must contend with attackers who can search, not just write.

Zou et al. 2023, arXiv:2307.15043

Instruction hierarchy

The instruction hierarchy teaches a model that instructions carry different privileges depending on their source: the System prompt outranks the developer, who outranks the user, who outranks retrieved or tool-generated text. Wallace et al. (2024) trained models to follow this ordering, mitigating Prompt injection by making the model likelier to ignore instructions arriving through low-privilege channels. It is a training-level mitigation, not an enforced boundary — all the channels still share one stream of Tokens.

Wallace et al. 2024, arXiv:2404.13208

Training data extraction

Training data extraction turns Memorization into a live leak: an attacker samples heavily from a deployed model, then filters the output for text reproduced verbatim from the training corpus. Carlini et al. (2020) demonstrated the attack against GPT-2, recovering names, contact details, and other Personally identifiable information (PII) with nothing more than ordinary query access. The result established that training data is not locked inside the model — anything memorized may be one well-chosen prompt away from disclosure.

Carlini et al. 2020, arXiv:2012.07805

Memorization

Memorization is when a model stores and can emit specific training sequences rather than just the patterns behind them. Carlini et al. (2022) quantified its drivers: memorization grows with model size, with how often a sequence is duplicated in the corpus, and with how much of the sequence is supplied as prompt context. The duplication finding is why Deduplication is a security control as much as an efficiency one, and memorized text is exactly what Training data extraction attacks go hunting for.

Carlini et al. 2022, arXiv:2202.07646

Membership inference

A membership inference attack asks a narrower question than extraction: was this particular example in the training data? The attacker typically compares the model’s confidence on a candidate text against its behavior on similar unseen text, exploiting the fact that models fit their training members more tightly. A positive answer can itself be sensitive — knowing someone’s records were in a medical model’s corpus reveals something about them — and the same confidence signals help Training data extraction attacks verify that what they recovered is genuine Memorization.

Personally identifiable information (PII) (Personally Identifiable Information)

PII is any data that can identify a specific individual — names, email addresses, phone numbers, government IDs. It enters LLM systems by two paths: scraped into training corpora, where Memorization can make it recoverable through Training data extraction, and pasted into prompts at inference time, where it may land in logs, caches, or model-improvement pipelines. Handling both paths — scrubbing corpora, redacting prompts, filtering outputs — is a baseline obligation under privacy regulations such as GDPR.

OWASP (Open Worldwide Application Security Project)

OWASP is the nonprofit whose Top 10 lists have long set the baseline vocabulary of web-application security. Its GenAI Security Project extends that model to LLMs: the OWASP Top 10 for LLM Applications ranks the risks practitioners most need to manage, with Prompt injection at the head of the list alongside entries like Excessive agency, System prompt leakage, and improper Output handling. The list gives teams a shared checklist — and shared names for failure modes that would otherwise be rediscovered one incident at a time.

OWASP GenAI Security Project, https://genai.owasp.org/llm-top-10/

Red teaming

Red teaming is structured adversarial testing: people (or attacker models) probe an LLM for Jailbreaks, policy violations, and harmful capabilities so that failures surface before deployment rather than after. Ganguli et al. (2022) described red-teaming language models at scale and released their attack transcripts, showing how the practice both measures harm and generates training data for defenses like Reinforcement learning from human feedback (RLHF). It complements automated attacks such as the Adversarial suffix: humans find new categories of failure, optimization finds their sharpest instances.

Ganguli et al. 2022, arXiv:2209.07858

Content filter

A content filter (or guardrail model) screens what enters or leaves an LLM: a classifier flags prompts before the model sees them, outputs before the user does, or both. Because it sits outside the model, it fails independently of the model’s own Refusal training — defense in depth against Jailbreaks that slip past Alignment. Filters trade coverage against false positives, and attackers probe them just as they probe the model, so a filter is one layer of Output handling discipline rather than a complete answer.

System prompt leakage

System prompt leakage is users extracting the System prompt — often by simply asking, or through Prompt injection. The OWASP LLM Top 10 frames the risk precisely: the leak itself is rarely the harm; the real vulnerability is having designed as though the prompt were secret — stashing credentials, role logic, or the whole security model in text the model can be talked into repeating. The defensive stance is to treat every system prompt as public and keep secrets and enforcement outside the model.

OWASP GenAI Security Project, https://genai.owasp.org/llm-top-10/

Excessive agency

Excessive agency is the OWASP LLM Top 10 entry for over-empowered models: too many tools, overly broad permissions, or too much autonomy relative to the task. The danger compounds with Indirect prompt injection — a hijacked model that holds an email tool and no confirmation step can act on an attacker’s behalf, not just talk. The mitigations are classic least privilege: a minimal tool surface, narrowly scoped credentials, and human approval for consequential actions.

OWASP GenAI Security Project, https://genai.owasp.org/llm-top-10/

Abstention

Abstention is the calibrated alternative to guessing: when the model’s knowledge runs out, it says so rather than emitting a confident Hallucination. It differs from Refusal, which declines on safety or policy grounds — abstention declines on epistemic ones. Post-training can encourage it, and Grounding gives the model a defensible basis for the answers it does give; but the tuning matters, because a model that abstains too readily is merely useless in a different way.

Output handling

Output handling is the discipline of treating LLM output as untrusted input to whatever consumes it. Model text that flows unexamined into a browser becomes XSS; into a shell, command injection; into SQL, the classic attack — the OWASP LLM Top 10 catalogs this failure as improper output handling. The rule inverts a common intuition: the model is not a trusted component, because anyone who can influence its Context window — including via Prompt injection — partially controls what it emits. Validate, encode, and sandbox accordingly.

OWASP GenAI Security Project, https://genai.owasp.org/llm-top-10/

Provisioned throughput

Provisioned throughput is dedicated capacity purchased from a managed model service: you commit to a term and the platform guarantees a Token-processing rate, insulating you from the shared on-demand pool and its Quotas. Amazon Bedrock sells it in Model Units billed hourly on no-commitment, 1-month, or 6-month terms — and a model customized in Bedrock must run on it — while Vertex AI sells fixed-term GSU subscriptions from 1 week to 1 year with overage spilling to pay-as-you-go, and Azure’s equivalent is the provisioned deployment bought in PTUs. It turns a variable per-token bill into a fixed cost, which only pays off when Utilization of the reserved rate stays high.

Amazon Bedrock User Guide, https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html

Managed endpoint

A managed endpoint is a deployed model exposed as an API the provider operates: you pick a model and a capacity level, the platform provisions machines, runs the Serving engine, and hands back a URL. Azure AI Foundry splits the idea into serverless deployments (Microsoft hosts the model and bills per Token) and managed compute (model weights on dedicated VMs billed by core-hours); Amazon Bedrock Marketplace models deploy to endpoints managed by SageMaker AI; Vertex AI Model Garden models can instead be self-deployed onto compute inside your own project. The trade is control for operations: against self-hosting on Kubernetes you give up tuning the stack and gain freedom from running it.

Azure AI Foundry model docs, https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/foundry-models-overview

Model catalog

A model catalog is the curated library of models a cloud platform will deploy or serve for you — first-party families, partner models, and Open weights entries side by side. Each platform brands its own: Amazon Bedrock’s Model Catalog lists serverless and Marketplace models in one place, Azure AI Foundry’s model catalog organizes Foundry Models into those sold by Azure and those from partners and community, and Google’s Model Garden groups Google, partner, and open models. The catalog entry is where a model’s options are stamped: whether it can be called serverlessly, deployed to a Managed endpoint, or customized with Fine-tuning.

Vertex AI Model Garden docs, https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/model-garden/explore-models

Quota

A quota is the platform-imposed ceiling on how much model traffic your account may send, metered in Tokens and requests per unit time. Azure assigns tokens-per-minute quota per subscription, region, model, and deployment type, with a proportional requests-per-minute limit and HTTP 429s beyond it; Bedrock controls inference through token-usage quotas with separate allocations per inference endpoint; Vertex AI enforces per-project, per-region, per-base-model request metrics. Quotas are why load tests surprise teams that only benchmarked latency: the remedies are an increase request, Provisioned throughput for guaranteed capacity, or Cross-region inference into other regions’ pools.

Azure AI Foundry quota docs, https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/quota

Capacity reservation

A capacity reservation trades money for certainty: you commit spend ahead of time and the provider guarantees the capacity will exist, whether that is raw GPU instances or managed model Throughput. Every platform sells a flavor — Bedrock’s Provisioned throughput and account-level Reserved tier, Azure’s provisioned deployments bought as provisioned throughput units (PTUs), Vertex AI’s fixed-cost, fixed-term Provisioned Throughput subscriptions. Reservations invert the cloud’s pay-for-what-you-use pitch into pay-for-what-you-reserved, so idle reserved capacity surfaces directly as waste in Utilization reviews; Spot capacity sits at the opposite end of the certainty spectrum.

Vertex AI consumption options, https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/deploy/consumption-options

Private endpoint

A private endpoint is a network interface with a private IP inside your own VPC or VNet that fronts a managed service, so calls to a Managed endpoint travel the provider’s backbone instead of the public internet. All three platforms document the pattern: AWS PrivateLink creates interface VPC endpoints for Bedrock, Azure Private Link remaps a Foundry resource’s DNS so in-network clients resolve a private IP, and Google’s Private Service Connect endpoints reach Vertex AI from VPC, on-premises, and multicloud networks. It is the standard answer to “prompts must never cross the public internet” — and often a hard requirement before production traffic is approved.

Amazon Bedrock User Guide, https://docs.aws.amazon.com/bedrock/latest/userguide/usingVPC.html

Cross-region inference

Cross-region inference lets a managed service route a request to a region other than the one you called, trading placement control for capacity when your home region’s Quota or availability runs short. Bedrock implements it as inference profiles — geographic profiles (US, EU, APAC) keep routing inside a geography for data residency, while global profiles may use any supported commercial region; Azure’s Global deployment types route to available datacenters in any region, with Data Zone types confined to a Microsoft-defined US, EU, or Asia Pacific zone; and Vertex AI offers a global endpoint that improves availability and reduces resource-exhausted (429) errors. The catch is residency: with global routing you cannot control which region processes a request, and Bedrock’s inference profiles do not currently support Provisioned throughput.

Amazon Bedrock User Guide, https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html

Spot capacity

Spot capacity is the provider’s spare compute sold at a deep discount, on the condition that it can be reclaimed with only minutes of warning. That suits interruptible, checkpointed work — batch scoring, evaluation sweeps, Fine-tuning runs that can resume — where a killed node costs a restart rather than an outage. Serving latency-sensitive traffic on spot is riskier: a reclaimed node drops every in-flight request and its KV cache, so production clusters typically cover baseline load with on-demand capacity or a Capacity reservation and let spot absorb the burst or batch tail.

Egress

Egress is data transfer out — of a region, an availability zone, or the provider’s network entirely — and it is the direction clouds meter and bill. For LLM systems the Token streams themselves are small; the egress that matters is moving model Weights, training corpora, and vector indexes between regions or providers, which is one reason true multicloud architectures are rarer than the diagrams suggest. Egress pricing creates data gravity: once your data and serving stack settle in one region, leaving carries a metered cost that belongs in any honest cost model.

Utilization

Utilization is the fraction of paid capacity actually serving work — busy GPU-hours over billed GPU-hours, or consumed versus reserved Throughput. It is the hinge of cloud cost modeling: pay-per-token pricing makes utilization the provider’s problem, while self-hosted clusters and Capacity reservations make it yours — a dedicated GPU idling overnight costs exactly as much as one at full load. Serving stacks raise it with batching and autoscaling; finance reviews chase it across reserved spend and Provisioned throughput commitments.

TPU (Tensor Processing Unit)

A Tensor Processing Unit is Google’s custom accelerator for neural-network workloads, designed in-house and offered through Google Cloud rather than sold as hardware. TPUs center on large matrix-multiply units and interconnect into pods for training and serving at scale — they are the silicon Google runs its own frontier models on. For customers self-hosting on GCP they are the main alternative to NVIDIA GPU instances: the economics can be attractive, but the software path runs through XLA-based toolchains such as JAX rather than the CUDA ecosystem most Serving engines target first, so support for your model and stack is the thing to check.

Trainium

Trainium is AWS’s custom accelerator for model training, the training-side counterpart to Inferentia. It powers EC2 Trn1 instances — the documented trn1.32xlarge carries 16 Trainium accelerators with 32 GB of accelerator memory each, 512 GB in total — offered as an alternative to NVIDIA GPU instances for training and Fine-tuning on AWS. Like Google’s TPU, it trades ecosystem for economics: workloads run through the AWS Neuron toolchain rather than CUDA, so framework support is the first thing to verify.

AWS EC2 accelerated computing instance specifications, https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html

Inferentia

Inferentia is AWS’s custom accelerator for inference, the serving-side counterpart to Trainium. Its second generation powers EC2 Inf2 instances, documented from inf2.xlarge with a single 32 GB accelerator up to inf2.48xlarge with twelve (384 GB total) — enough to host mid-sized Open weights models, especially with Quantization. As with Trainium, models compile through the AWS Neuron toolchain rather than CUDA, so confirm your architecture and Serving engine are supported before planning capacity around it.

AWS EC2 accelerated computing instance specifications, https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html

Model family

A model family is a developer’s named lineage of models released in multiple sizes and successive versions — Meta’s Llama line spans Open weights releases from 1B-parameter variants to Llama 3.1’s documented “8 billion to 405 billion parameters”. Families, not individual checkpoints, are the useful unit for mapping the landscape: sizes and version numbers churn quarterly, while a family’s developer, license pattern, and release habits stay comparatively stable. Choosing a family first and a size second is how practitioners keep a mental map that survives the next Model release.

Meta Llama organization on Hugging Face, https://huggingface.co/meta-llama

Model release

A model release is the unit by which the landscape changes: a versioned artifact — downloadable Open weights or a new Closed weights API model — shipped with a license and, normatively, a Model card documenting intended use and evaluations. Releases arrive as collections rather than single checkpoints: one announcement typically covers several sizes and variants of a Model family. What a release actually includes varies enormously — SmolLM3’s card advertises a “Fully open model: open weights + full training details including public data mixture and training configs”, while most releases publish weights alone, or nothing but an endpoint.

Mitchell et al. 2018, arXiv:1810.03993

Deprecation

Deprecation is a provider signaling that a model version is on its way out — new work should move off it before it is retired. It is routine landscape churn: Mistral’s official docs keep a deprecated/retired table where whole earlier families (Mixtral, Magistral, Devstral, Pixtral) now sit, and OpenAI’s model page marks entries like GPT-Realtime Mini as deprecated. For Closed weights models, deprecation eventually means the endpoint disappears; with Open weights you can pin the downloaded checkpoint and keep serving it — one of the strongest arguments in the open-versus-closed decision.

Mistral models overview, https://docs.mistral.ai/getting-started/models/models_overview/

Community license

A community license is a vendor-drafted license attached to some Open weights releases: it grants broad use rights but adds custom conditions that standard permissive licenses do not carry. The Llama 4 Community License Agreement requires “Built with Llama” attribution, derived model names starting with “Llama”, and a separate license for entities exceeding 700 million monthly active users. That places community-licensed models between permissively licensed releases (Apache 2.0, MIT) and Closed weights APIs — the weights are downloadable, but the terms need legal review before commercial use.

Llama 4 Community License Agreement, https://developer.meta.com/ai/llama4/license/

Gated distribution

Gated distribution publishes model weights behind an acceptance step: the artifact is hosted publicly, but the download unlocks only after you agree to the license. Meta’s Llama models on Hugging Face are the canonical case — weights become downloadable after you “accept the license terms and acceptable use policy”. The gate is what makes a Community license enforceable at distribution time, and it is one reason Open weights is not a synonym for open source: published and unconditionally available are different things.

Meta Llama organization on Hugging Face, https://huggingface.co/meta-llama

Multimodal

A multimodal model handles more than one Modality — most commonly images in addition to text on the input side, with text out. Coverage varies widely and is documented per model: Gemma 3 is documented as “handling text and image input and generating text output”, while Qwen2.5-Omni-7B is documented to “perceive diverse modalities, including text, images, audio, and video” while generating text and speech. Under the hood, a Vision encoder (or its audio equivalent) converts the extra modality into representations the Large language model (LLM) backbone can attend over.

Qwen2.5-Omni-7B model card, https://huggingface.co/Qwen/Qwen2.5-Omni-7B

Modality

A modality is one kind of data — text, image, audio, video — that a model can take as input or produce as output. The two directions rarely match: Llama 3.2-Vision’s card lists input modalities as “Text + Image” but output as text only, and most Multimodal models today widen inputs while still emitting only text. Reading a Model card’s input and output modality lines separately is the fastest way to learn what a model actually does.

Llama 3.2 Vision model card, https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct

Vision encoder

A vision encoder converts images into sequences of Embeddings that the language backbone can process alongside text tokens. Gemma 3’s card documents the mechanics concretely: images are “normalized to 896 x 896 resolution and encoded to 256 tokens each”, while Phi-4-Reasoning-Vision pairs a SigLIP-2 vision encoder with its language model in a mid-fusion architecture. Because encoded images occupy sequence positions, heavy image input consumes the Context window like any other tokens.

Gemma 3 model card, https://huggingface.co/google/gemma-3-4b-it

Small language model (SLM) (Small Language Model)

A small language model is an Large language model (LLM) at a size chosen for deployability rather than peak capability — there is no fixed cutoff, but documented examples run from Qwen3-0.6B (“Number of Parameters: 0.6B”) through SmolLM3’s “3B parameter language model” to Gemma 3’s 1B and 4B sizes. Small models increasingly inherit big-model features: SmolLM3 documents “dual mode reasoning, 6 languages and long context”, and Qwen3-0.6B switches between thinking and non-thinking modes in one model, echoing a Reasoning model at a fraction of the size. With Quantization — and often Distillation from larger teachers — SLMs are the models that make Edge deployment practical.

SmolLM3-3B model card, https://huggingface.co/HuggingFaceTB/SmolLM3-3B

Edge deployment

Edge deployment runs a model on end-user hardware — phones, laptops, embedded devices — rather than behind a hosted API. Vendors now target it explicitly: Gemma 4’s E2B and E4B variants are pitched for “mobile and IoT devices” with larger sizes offering “Frontier intelligence on personal computers”, and OpenAI documents gpt-oss-20b for lower-latency local and specialized use. The gains are privacy, offline operation, and zero marginal token cost; the constraint is device memory, which is why the edge leans on a Small language model (SLM) plus aggressive Quantization — and requires Open weights to begin with.

Google DeepMind Gemma models page, https://deepmind.google/models/gemma