The learning path

Stage 1: First principles

What a language model is and what it can honestly do.

What Is a Large Language Model? — The one-sentence definition unpacked: a next-token predictor, not a database — and the vocabulary you need before anything else makes sense.
Next-Token Prediction, Really — The single objective behind everything an LLM does: a probability for every token, a measurable notion of being wrong, and how prediction becomes ability.
From N-grams to Transformers — Seventy years of language modeling in one arc: counting words, the neural turn, attention, and the scale era — with the dates and papers that mattered.
Capabilities and Hard Limits — What LLMs do well, what they cannot do by construction, and why the frontier is jagged — testing beats trusting.
The LLM Lifecycle Map — Pre-training, post-training, inference, adaptation: the map this whole academy is organized around, and who does what at each stage.
In Production: Where the Meters Run — The domain’s production capstone: cost, latency, reliability, and security as the four meters every later domain reads — and where each one lives in the stack.

Stage 2: The machine, opened up

Tokens, attention, and the transformer block — how the model actually computes.

Tokenization: Text Becomes Tokens — Why models read subword tokens instead of words, how byte-pair encoding builds a vocabulary, and the artifacts every engineer eventually hits.
Embeddings: Tokens Become Vectors — The embedding matrix, what geometry in representation space means, contextual versus static embeddings, and the unembedding back to logits.
Positional Encoding: Teaching Order to a Set — Attention is order-blind by construction. Sinusoidal and learned positions, RoPE, and how position schemes shape context extension.
Self-Attention: Every Token Looks at Every Token — Queries, keys, and values; the scaled dot-product equation term by term; causal masking — the mechanism that made the transformer.
Multi-Head, Multi-Query, Grouped-Query Attention — Why many small heads beat one big one, what heads actually learn, and how MQA/GQA trade attention quality for serving memory.
Inside the Block: FFN, Residuals, Normalization — The other half of the transformer block: feed-forward networks, the residual stream, and why normalization placement matters.
Mixture of Experts: Sparse by Design — Routers, experts, and load balancing — how MoE models activate a fraction of their parameters per token, and what that costs in memory.
One Token's Journey: The Full Forward Pass — The whole stack assembled end to end — from input text to next-token probabilities, and where the parameters actually live.
Architecture in Production: Design Choices Set Your Bill — The domain's production capstone: how attention's quadratic tax, GQA, MoE, and context length decisions surface as cost, latency, and capacity on real cloud hardware.

Stage 3: How models are made

Pre-training at scale, then the post-training that shapes behavior.

Data Pipelines and Curation — Where training text actually comes from, and the pipeline that turns a crawl into a corpus: filtering, deduplication, and the mixture decisions that shape a model.
Objectives and the Loss Curve — Causal language modeling and its rivals — masked and span objectives — plus how to read the one chart every training run lives by.
Scaling Laws — The discovery that loss follows the budget: Kaplan’s power laws, Chinchilla’s compute-optimal correction, and why modern models overtrain on purpose.
Compute Budgets — From FLOPs accounting to GPU-hours: how training budgets are actually estimated, and where the money goes.
Distributed Training — No single device fits the model or the data: data, tensor, and pipeline parallelism — and how real runs combine all three.
Mixed Precision and Stability — FP32, FP16, BF16 — why training runs in mixed precision, and what loss spikes, checkpoints, and restarts look like at scale.
In Production: Should You Ever Pre-train? — The domain’s production capstone: the honest decision framework for training from scratch versus continued pre-training versus not doing this at all — and what a training cluster demands.
From Base Model to Assistant — A base model completes text; an assistant holds a conversation. The post-training map: what each stage adds, and what actually changes inside the model.
Supervised Fine-Tuning — Teaching by demonstration: instruction data, chat templates, the training recipe — and why data quality beats quantity.
RLHF: Learning from Preferences — Reward models trained on human comparisons, policy optimization against them, and the failure mode that haunts the whole method: reward hacking.
DPO and the Direct Methods — Direct preference optimization skips the reward model entirely. The trick, the math, the variants — and when classic RLHF still wins.
AI Feedback: RLAIF and Constitutional Methods — When the annotator is a model: AI-generated preferences, constitutions and critique loops, and the honest limits of self-supervision.
RL for Reasoning Models — Verifiable rewards changed the game: training models to think longer on math and code, what reasoning training buys, and what it costs at inference time.
In Production: What Alignment Does and Doesn’t Guarantee — The domain’s production capstone: the behavior profile you inherit with every model, the gap between aligned and safe-for-your-app, and how to test before you trust.

Stage 4: Running the model

Inference, serving, and the metrics that decide your latency and bill.

Prefill and Decode — One request, two very different phases: the parallel prefill and the serial decode — the split behind every latency number and price sheet in this field.
Decoding and Sampling — Greedy, beam, temperature, top-k, top-p: how one token actually gets chosen, and how to pick settings you can defend.
The KV Cache — The memory that makes generation affordable: what gets cached, what it costs, paged attention, and prefix reuse.
Context Windows and Long Context — The window as a budget: what long context really costs, the techniques that stretch it, and why the middle of a long prompt is a bad neighborhood.
Quantization — Fewer bits, same model — mostly: what gets quantized, the methods that made it routine, and how to check what it cost you.
Speculative Decoding — Draft cheap, verify exact: how a small model makes a big one faster without changing a single output token.
Continuous Batching and Serving Engines — Why a GPU serving one request at a time is a tragedy, how continuous batching fixes it, and a map of the engines that do this for a living.
In Production: TTFT, TPOT, and the Metrics That Matter — The domain’s production capstone: the latency vocabulary, the throughput–latency frontier, and how serving choices surface on the three clouds.

Stage 5: Making it yours

Prompting, RAG, and fine-tuning — adaptation without illusions.

Prompting as Engineering — The cheapest, most reversible adaptation lever: instructions, examples, and structure — treated like code, not incantations.
Context Engineering at the Model Level — The window is a budget you allocate: what goes in, where it sits, and how structure interacts with caching and attention.
RAG Mechanics: Embeddings and Search — The machinery under retrieval: embedding models, chunking, approximate nearest-neighbor search, and hybrid ranking.
RAG End to End — From ingest to grounded answer: the full pipeline, reranking, the failure modes that actually occur, and how to evaluate any of it.
Fine-Tuning: Full, LoRA, QLoRA — Moving behavior into the weights: full fine-tuning, low-rank adapters, quantized training — and the recipe that keeps it honest.
Distillation — Teaching a small model with a big one: synthetic data, filtering, and what actually transfers.
In Production: Choosing Your Lever — The domain’s production capstone: prompt, retrieval, or weights — the decision framework, the cost shapes, and why real systems compose all three.

Stage 6: Trust but verify

Evaluation and the model-level risk map.

Benchmarks and Their Limits — What a benchmark actually measures, how benchmarks saturate and get gamed, and how to read a leaderboard number without being fooled.
Data Contamination — When the test set leaks into the training set: why contamination happens at web scale, how it is detected, and what a score means once it has.
Human Eval, Arenas, and LLM Judges — When there is no answer key: human evaluation, preference arenas and their ratings, and models judging models — with the biases of each.
Building an Eval Harness — Your benchmark, for your task: golden sets from real traffic, the grader ladder, and a harness you can rerun on every change.
Production Evals and Regression Testing — Evals as the gate: regression testing before any model or prompt change ships, quality monitoring after, and surviving vendor model updates.
In Production: Reading Model Evals Critically — The domain’s production capstone: how to read a vendor eval table, why cross-vendor comparisons mislead, and why your own numbers always win.
Hallucination — Why models confabulate by construction, the shapes it takes, what actually reduces it — and how to measure it instead of hoping.
The Single-Channel Problem — The root vulnerability: one token stream with no type system separating instructions from data — and why training alone cannot close it.
Prompt Injection and Jailbreaks: The Model-Level View — Direct and indirect injection, the jailbreak taxonomy, and why refusals are trained dispositions rather than enforcement.
Data Leakage and Memorization — Models memorize training text and can be made to emit it; prompts flow to providers; deletion is hard. The leakage map, both directions.
The OWASP Top 10 for LLM Applications — The community-standard risk list for LLM applications, walked entry by entry and mapped onto this site — plus how to actually use it.
Red Teaming LLMs — Adversarial probing as a discipline: manual and automated methods, coverage, and turning findings into fixes.
In Production: The Defense Stack Around a Served Model — The domain’s production capstone: defense in depth around a model that cannot defend itself — provider filters, app-side controls, and monitoring.

Stage 7: Production synthesis

Clouds, costs, architectures, and choosing models like an engineer.

Managed APIs vs Self-Hosting — The consumption spectrum from serverless tokens to your own GPU fleet: what you inherit at each step, and the drivers that actually decide it.
Bedrock, Azure AI Foundry, Vertex AI — The three managed model platforms share one shape — catalog, serverless, provisioned, customization, guardrails. The tour, verified against the docs.
Self-Hosting on EKS, AKS, GKE — Running your own serving stack on managed Kubernetes: the GPU node reality, the engine deployment, and the day-2 work nobody budgets.
GPUs and Accelerators — The hardware menu across the three clouds — what the specs actually mean for LLM serving, and the custom-silicon alternatives.
Cost Modeling — Three meters — per-token, provisioned units, GPU-hours — and the break-even thinking that decides between them, with utilization as the hidden variable.
Reference Architectures — Three architectures that cover most LLM systems — the managed-API app, the RAG stack, the self-hosted cluster — drawn end to end, per cloud.
In Production: Multi-Region, Quota, and Failover — The domain’s production capstone: quotas as an architectural force, cross-region serving mechanisms, and failover patterns that survive contact with reality.
Model Families — The landscape organized the way it actually moves: families, not models — who builds them, how they release, and how to keep a map that survives next quarter.
Open vs Closed Weights — “Open” is a spectrum, not a switch: what weights availability actually grants, what the license fine print takes back, and the trade that decides deployments.
Multimodal Models — Images, audio, and video join the token stream: what multimodal actually means inside the model, and what it changes at the meter.
Small Language Models — The counter-trend that matters: why small models keep winning real workloads, who builds them, and how small gets made good.
Reasoning Models in the Landscape — Thinking became a product tier: reasoning variants across the families, the documented effort dials, and when to pay for thought.
In Production: Reading a Model Card and Choosing a Model — The domain’s production capstone — and the site’s closing skill: the model-card anatomy, the shortlist protocol, and staying current without chasing.