Certification Academy glossary

831 terms with primary sources.

MLA-C02

The second version of the AWS Certified Machine Learning Engineer – Associate exam. Announced July 2026, beta from September 29, 2026, standard delivery announced for January 14, 2027. Compared with MLA-C01 it adds foundation-model development, RAG, agent deployment, and GenAI operations (Amazon Bedrock and Bedrock AgentCore are in scope), and drops model pruning/compression, SageMaker Neo, and BYOC depth. AWS's certification page briefly printed the beta code as "ME1-C02" — same exam.

AWS Training & Certification, October 2026

MLA-C01

The first version of the ML Engineer – Associate exam: 65 questions, 130 minutes, $150, weights 28/26/22/24, and the debut of ordering/matching/case-study question types (dropped again in C02). Continues in Japanese, Korean, and Simplified Chinese until MLA-C02 reaches standard delivery.

AWS Training & Certification, October 2026

Scaled score

AWS converts your raw performance to a 100–1,000 scale so that scores are comparable across different exam forms of varying difficulty. The associate pass line is 720. Because of equating, 720 does not literally mean 72% of questions answered correctly.

Compensatory scoring

The MLA exam guide states candidates "do not need to achieve a passing score in each section" — only the total scaled score counts. Strategic consequence: spend marginal study time where weight × weakness is largest, and never write off a weak domain as fatal.

Multiple response

One of the two MLA-C02 question formats (with multiple choice). The stem states how many options to select. Grading is all-or-nothing: selecting one right answer of two scores zero, so completing the set with a best guess strictly dominates leaving it short.

Unscored questions

AWS mixes 15 experimental items into the standard 65-question form to calibrate future exams. They are indistinguishable from scored items, so every question deserves full effort. (The 85-question beta form does not follow the 50+15 split.)

Exam blueprint

The content outline in the official exam guide: Domain 1 Data Preparation for ML and AI (28%), Domain 2 ML Model and FM Development (24%), Domain 3 Deployment and Orchestration (24%), Domain 4 Operating, Monitoring, and Securing (24%) — each broken into task statements that name the skills and services in scope.

Amazon SageMaker AI

The managed platform for the full ML lifecycle: notebooks and Studio, Data Wrangler and Feature Store for preparation, training jobs with automatic model tuning, a Model Registry for versions, managed MLflow for experiments, and real-time/batch inference hosting. Appears in every MLA exam domain. (Renamed from "Amazon SageMaker" to "Amazon SageMaker AI" in late 2024.)

Amazon Bedrock

The managed foundation-model service: choose among hosted FMs, customize them (fine-tuning, continued pre-training, Custom Model Import), ground them with Knowledge Bases (RAG), constrain them with Guardrails, evaluate them, and run agents — with Bedrock AgentCore providing agent runtime and observability. Central to the GenAI half of MLA-C02.

AWS Skill Builder

The official AWS training platform. For MLA-C02 it carries a 4-step exam prep plan, the Official Practice Question Set, Official Pretest, and Official Practice Exam, plus Builder Labs, Cloud Quest, and SimuLearn for hands-on practice. Free tier covers much of the prep plan; labs and the practice exam sit behind the subscription.

ML lifecycle

The mental map the MLA-C02 blueprint is organized around: data is collected and prepared (Domain 1), a model is trained and evaluated (Domain 2), deployed (Domain 3), and operated (Domain 4) — and what monitoring finds (drift, decay) feeds the next retraining lap. Placing a scenario question on this loop narrows the candidate services before reading a single option.

SageMaker Studio

The browser-based development environment of Amazon SageMaker AI — notebooks and code editors plus entry points into Data Wrangler, training jobs, experiments, and deployments. Home directories are backed by shared file storage, so work persists across sessions.

SageMaker Feature Store

One feature pipeline feeding both halves of the lifecycle: the online store serves low-latency feature lookups at inference time, and the offline store builds point-in-time-correct training sets. Centralizing features ends train/serve skew and lets teams reuse each other's engineering — the exam cue is "share features across teams, online and offline".

Automatic model tuning (AMT)

Managed hyperparameter search on SageMaker AI. You declare the tunable ranges and the objective metric; AMT coordinates a series (or parallel sets) of training jobs using Bayesian optimization, random search, Hyperband, or grid search, and reports the best-performing configuration.

SageMaker Model Registry

The versioned catalog of trained models on SageMaker AI: model packages grouped by use case, each version carrying metadata, lineage, and an approval status. Deployment pipelines gate on that status, so "only approved models reach production" questions resolve here.

Managed MLflow

SageMaker AI operates the MLflow tracking server for you: experiments and runs with parameters, metrics, and artifacts, comparable across attempts, with registration flowing toward deployment. The exam cue is "track and compare experiment runs" without standing up tracking infrastructure.

Foundation model (FM)

The model class behind the GenAI half of MLA-C02: pre-trained at scale, then adapted — prompt engineering, retrieval-augmented generation, fine-tuning, continued pre-training — instead of being built per task. On AWS, hosted FMs live behind the Amazon Bedrock API; self-managed FMs run on SageMaker AI.

Retrieval-augmented generation (RAG)

The default fix for "the model hallucinates about our domain": embed the query, search a vector store for the nearest document chunks, and generate with those chunks in context — often with citations. On AWS the managed implementation is Bedrock Knowledge Bases; the exam contrasts RAG (fresh, source-grounded, no training) with fine-tuning (behavioral change baked into weights).

Bedrock Knowledge Bases

Amazon Bedrock's managed retrieval-augmented generation: connect data sources (S3 and others), and the service chunks and embeds the documents into a vector store it manages (Amazon OpenSearch Serverless by default, or one you bring), then retrieves the best chunks per query so the model answers grounded in your content. Exam cue: "reduce hallucination with company documents".

Bedrock Guardrails

Policy as configuration rather than a custom moderation model: content filters (including prompt-attack screens), denied topics, word filters, sensitive-information (PII) handling, and contextual-grounding checks, enforced on both prompts and responses. The ApplyGuardrail API extends the same policy to models outside Bedrock. Exam cues: "block harmful prompts", "mask PII in responses".

Bedrock AgentCore

Amazon Bedrock AgentCore operates agents in production regardless of the framework they were built with: a serverless runtime, supporting services (session memory, identity, a gateway that turns APIs into agent tools), and observability dashboards tracing every step. New to the exam in MLA-C02, which added agent deployment to the blueprint.

Custom Model Import

The Bedrock path for teams that trained or fine-tuned an open-weights model elsewhere (supported architectures) and want serverless serving without managing instances: import the weights, invoke through the standard Bedrock API alongside the hosted models. The tiebreaker against SageMaker hosting is operational: Bedrock for no-infrastructure serving, SageMaker when you need control of the serving stack.

AWS AI services

The highest altitude on the AWS ML stack: Textract (documents/forms), Rekognition (images/video), Comprehend (text analysis incl. PII), Transcribe (speech-to-text), Translate, Polly (text-to-speech), Lex (conversational bots), Personalize (recommendations). Several accept light customization (custom vocabularies, entities, labels). When a scenario describes a commodity task and no ML appetite, these beat building anything.

Glue Data Catalog

The central metadata registry of the lake: crawlers scan data sources, infer schemas, and write table definitions here; Athena queries against it, EMR and Redshift Spectrum read it, and Lake Formation attaches table- and column-level permissions to it. Exam cue: "catalog and crawl data in S3".

AWS Step Functions

Orchestration as a managed state machine: steps call AWS services directly (SageMaker training jobs, Bedrock, Lambda, Glue), with built-in retries, catch/branch logic, waits, and human-approval patterns. The reflex answer to "multi-step ML workflow with retries" — distinguished from EventBridge (which triggers workflows) and MWAA (Airflow DAGs for teams already on Airflow).

Amazon EventBridge

EventBridge connects "something happened" to "do something": rules match events from AWS services (a CloudWatch alarm state change, a new S3 object) or fire on schedules, then invoke targets — state machines, Lambda functions, queues. In MLA scenarios it is the trigger in front of retraining pipelines: alarm → rule → Step Functions.

Requirement qualifier

MLA distractors usually work; the qualifier decides which working option is the key. "MOST cost-effective" votes for Spot, batch, and serverless; "LEAST operational overhead" votes for managed and serverless; "HIGHEST performance" votes for provisioned, distributed, and GPU-backed. Reading the qualifier before the options is the single highest-yield exam habit.

Apache Parquet

The default analytical format of the S3 data lake. Values are stored column-by-column in compressed chunks with min/max statistics, so Athena, Glue, and EMR read only the columns a query names and skip chunks that cannot match a filter — unlike row formats (CSV, JSON), where every byte of every row is read. Internally compressed (e.g. Snappy) yet still splittable across readers, which gzipped CSV is not. ORC occupies the same niche with Hive-leaning tooling.

Amazon Kinesis Data Streams

The AWS-native streaming backbone. Producers write records with a partition key (ordering is preserved per key within a shard); multiple independent consumers (KCL apps, Lambda, Managed Flink) read the same stream; and the stream can be re-read anywhere within its retention window — the replay property Firehose lacks. Choose it when stems mention several consumers, strict per-key order, or reprocessing. MSK (managed Kafka) answers the same needs for Kafka-native teams.

Amazon Data Firehose

The easiest road from a stream to the lake: near-real-time delivery (it buffers by size or time), optional in-flight transformation via a Lambda function, and built-in record format conversion to Parquet/ORC using a Glue Data Catalog schema. It delivers rather than stores — no replay, no consumer management. Renamed from "Amazon Kinesis Data Firehose" in 2024; older material uses the old name.

Amazon Managed Service for Apache Flink

Runs Apache Flink applications (SQL, Java, Python) against streams from Kinesis or MSK with managed, checkpointed state — the service for computations that span records and time: sliding-window counts, sessionization, stream-to-stream joins, with event-time watermarks for late data. It processes streams; it does not store them. The canonical ML pattern: Kinesis → Flink computes rolling features → SageMaker Feature Store online store.

Online store (Feature Store)

Holds only the most recent value per record identifier, optimized for single-record reads at request latency. Endpoints call GetRecord during scoring so inference uses exactly the values the feature pipeline wrote — the serving half of the skew-prevention story. Not a historical or analytical store: no time travel, no bulk scans; that is the offline store’s job.

Offline store (Feature Store)

Every ingested feature value lands here with its event time, building an append-only history in your S3 bucket. Tables register in the Glue Data Catalog, so Athena (or Spark) assembles training datasets — including point-in-time ("time travel") queries that reconstruct what was knowable at each historical prediction moment, the honest defense against leaking future values into training.

Training/serving skew

The classic version: training computes a feature in warehouse SQL while the inference service re-implements it in application code; a rounding rule or window boundary differs, and the model scores production traffic on inputs it never saw in training. Offline metrics stay beautiful, production disagrees. The architectural cure is computing each feature once and serving both sides from the same record — the feature store’s founding purpose.

Vector database

Where RAG retrieval happens: embeddings live under an approximate-nearest-neighbor index (typically HNSW) that answers "closest vectors to this query" in milliseconds, with metadata filters narrowing the candidates. Exam framing is match-the-store: OpenSearch Service/Serverless for large-scale, low-latency hybrid search (and the Bedrock Knowledge Bases default); pgvector when vectors belong next to relational rows; S3 Vectors when scale and cost dominate and queries are infrequent.

Embedding

An embedding model maps content into a high-dimensional vector space where semantic similarity becomes geometric proximity. In RAG prep, every chunk is embedded at index time and the question is embedded with the same model at query time — mixing models breaks the space. Multimodal embedding models (e.g. Titan Multimodal Embeddings on Bedrock) place text and images in one space, enabling cross-modal retrieval. Embeddings are not reversible redaction: PII embedded is PII retained.

Chunking

The data-prep decision that bounds what RAG retrieval can ever return. Fixed-size chunks with 10–20% overlap are the predictable default; semantic chunking cuts where meaning shifts; hierarchical chunking matches small child chunks but returns the larger parent (long structured documents); and already-small units (FAQ entries) need no chunking. Bedrock Knowledge Bases supports all four plus custom chunking via Lambda. Symptoms map back: precise-but-contextless answers → chunks too small or no parent; diluted matches and token burn → chunks too big.

AWS Glue

The data-engineering workhorse of Domain 1: serverless Spark jobs in PySpark/Scala (or visual DAGs in Glue Studio) that extract from JDBC sources, join and clean, and write partitioned Parquet back to the lake — all registered in the Glue Data Catalog. The family includes crawlers (schema/partition inference), Glue Data Quality (DQDL rule gates), sensitive-data detection (Detect PII transform), and DataBrew. The exam’s shorthand: coded Spark ETL with the least operational overhead.

AWS Glue DataBrew

Point-and-click data preparation for people who will not write Spark: transformations compose into recipes that run as jobs, and profile jobs produce dataset statistics — missing-value rates, cardinality, distributions, correlations, outliers — with zero code. In the exam’s tool fork it is the "business analyst, no code" answer, versus Glue (engineers, code), EMR (cluster control), and Data Wrangler (ML-focused, inside Studio).

SageMaker Data Wrangler

The data-scientist member of the transform toolkit: import from S3/Athena/Redshift and more, build transform flows visually, and read built-in data-quality insights and Clarify-powered bias reports before training. Its differentiator is where it exports: a SageMaker Pipelines step, a Feature Store ingestion, a Processing job, or generated notebook code. Exam tell: "data scientist in Studio preparing features" with an ML destination.

Amazon Macie

Macie answers "where does sensitive data live in our lake?" at bucket scale: it inventories S3, applies managed and custom data identifiers, and emits findings for buckets and objects containing PII and other sensitive types. In ML data prep it is the discovery step — find it with Macie, then scrub it in text with Comprehend or in-pipeline with Glue’s Detect PII transform. Macie locates; it does not transform your data.

AWS Glue Data Quality

Quality as an enforceable contract: rules written in DQDL (Data Quality Definition Language) — RowCount, Completeness, ColumnValues, Uniqueness and friends — are evaluated against a dataset, producing per-rule verdicts and an overall score. Runs against Data Catalog tables (standing health checks) or inside Glue ETL jobs, where results can quarantine failing rows, fail the job before bad data lands, or alert via EventBridge/CloudWatch. Glue can recommend a starter ruleset by profiling the data.

SageMaker Ground Truth

Turns unlabeled S3 objects into training data: a labeling job pairs a task UI (boxes, classes, custom) with a workforce — the public crowd (Mechanical Turk), vendors from AWS Marketplace, or a private workforce for confidential or expert work. Several workers label each item and answers consolidate into one label with confidence. Automated data labeling trains on labels-so-far and auto-labels confident items, routing only the uncertain to humans. Distinct from Amazon A2I, which human-reviews model predictions in production.

Data leakage

The exam’s favorite data trap, in four flavors: target leakage (a feature recorded after the outcome — amount_refunded predicting fraud); temporal leakage (random splits of time-series let training see the future); preprocessing leakage (scalers/encoders/imputers fit on the full dataset before splitting); duplicate leakage (identical or near-identical rows on both sides, including oversampling before the split). The symptom is always "brilliant offline, baffling in production"; the diagnostic question is "would this value exist at prediction time?"

SMOTE (Synthetic Minority Oversampling Technique)

A rebalancing technique for class imbalance: rather than copying minority rows (naive oversampling), SMOTE synthesizes new points along the lines between a minority sample and its nearest minority neighbors in feature space. Applied after splitting and only to the training set — synthesizing before the split spreads near-copies across the train/test boundary (duplicate leakage) and inflates recall. Alternatives on the same shelf: undersampling, class weights, threshold tuning.

SageMaker Clarify

Clarify spans the lifecycle, but its Domain 1 role is auditing data before any model exists: pre-training bias metrics such as class imbalance (CI) and difference in proportions of labels (DPL), computed across a facet like gender or age and surfaced in Data Wrangler’s bias report. Later domains meet it again for post-training bias, feature attributions (SHAP), and explainability — same service, different checkpoints.

AWS Lake Formation

Where IAM and bucket policies stop (bucket/prefix granularity), Lake Formation continues: permissions on Data Catalog resources down to columns, row-level security via data filters, and tag-based access control (LF-Tags) that scales — classify PII columns once, grant "everything not tagged PII" broadly. One central permission model enforced by the engines that read the lake, which is how an ML team gets feature columns without identity columns, scoped to its own rows.

Amazon Comprehend

The free-text half of the privacy toolkit: where Macie discovers sensitive data at S3 bucket scale, Comprehend analyzes the text itself — real-time or batch — returning PII entity types (names, SSNs, bank accounts, and more) with confidence scores and character offsets. Redaction, replacing entities in the output text, runs as an asynchronous batch job. The canonical ML-prep pairing the exam tests: Macie to find which objects hold PII, Comprehend to scrub transcripts and tickets before they feed embeddings or fine-tuning.

SageMaker built-in algorithms

SageMaker AI ships a library of pre-implemented algorithms you launch as training jobs without writing model code: XGBoost and Linear Learner for tabular prediction, K-Means for clustering, Random Cut Forest for anomaly detection, DeepAR for probabilistic forecasting, and more. You point them at S3 channels, set hyperparameters, and pick instances. When one matches the problem shape, it is the lowest-effort way to train on your own data — the tradeoff is little control over the architecture.

Script mode

The middle rung of SageMaker training: you supply a training script as the entry point to a framework container (PyTorch, TensorFlow, Hugging Face, XGBoost, scikit-learn), and SageMaker runs it as a managed training job — channels, metrics, checkpoints, and artifact handling included. You own the model logic; AWS owns the fleet. Step down to a fully custom container only when exotic dependencies demand it (a topic MLA-C02 de-emphasizes).

Training job

The unit of model training on SageMaker AI. You declare the algorithm container, S3 input channels, instance type and count, hyperparameters, and an output path; SageMaker provisions instances, mounts data under /opt/ml/input/data/, streams logs and metrics to CloudWatch, uploads /opt/ml/model as model.tar.gz to S3, and tears the fleet down. Everything else in Domain 2 — tuning, Spot, distribution — decorates this contract.

Managed Spot Training

SageMaker training on spare EC2 capacity at a steep discount over on-demand. Spot capacity can be reclaimed, so jobs must tolerate interruption: enable checkpointing to S3 so a restarted job resumes rather than starting over, and set a maximum wait time at least as long as the maximum runtime. The exam cue is "minimize training cost" plus "the job can tolerate interruptions" or a flexible deadline.

Distributed training

Spreading one training run across GPUs or instances. Data parallelism replicates the model on every device and shards each batch, synchronizing gradients — the default when the model fits in one GPU's memory but training is slow. Model parallelism splits the model itself across devices when its parameters and activations cannot fit on one. SageMaker provides managed libraries for both, and script mode runs the frameworks' native options too.

Overfitting

High variance: the model fits training noise, so training loss keeps improving while validation loss stalls or rises. Remedies: more or augmented data, regularization (L1/L2, dropout), early stopping, a simpler model, or ensembling. Diagnose it from the train/validation gap — never from training metrics alone.

Underfitting

High bias: the model lacks the capacity or the training to capture the signal, so both training and validation performance are poor. Remedies: a bigger or more expressive model, better features, longer training, a higher learning rate, less aggressive regularization — the opposite medicine from overfitting, which is why diagnosis comes first.

Regularization

The family of techniques that constrain a model to fight overfitting: L1 (drives weights to zero, giving sparsity) and L2 (shrinks weights) penalties, dropout (randomly silencing units during training), early stopping on validation loss, and data augmentation. Exam-level skill: recognize these as overfitting remedies — and never prescribe them for an underfit model.

Ensemble learning

Combining models to beat any single one. Bagging trains independent models on bootstrap samples in parallel and averages or votes (Random Forest) — it reduces variance. Boosting trains sequentially, each learner correcting its predecessors' errors (XGBoost) — it reduces bias. Stacking feeds diverse base models' predictions into a meta-model. The costs: more compute and less interpretability.

Prompt engineering

The zero-training customization layer: system prompts set role and rules, zero-shot relies on instructions alone, few-shot adds worked examples, and structured-output prompting constrains format (for example, JSON against a schema). Instant to iterate and cheapest to run — but it cannot add private knowledge (that is RAG) or durably reshape behavior at scale (that is fine-tuning).

Fine-tuning

Continuing an FM's training on labeled prompt-response pairs so behavior, tone, or format moves into the weights. Available managed on Amazon Bedrock (select models; the result is a private custom model) and on SageMaker AI for open-weight models. Full fine-tuning updates every weight; parameter-efficient methods train small added weights with the base frozen — far cheaper, with lower catastrophic-forgetting risk.

Catastrophic forgetting

The classic fine-tuning failure: training hard on a narrow dataset overwrites general capabilities the base model had. Mitigations: mix general-purpose examples into the tuning data, use a lower learning rate and fewer epochs, prefer parameter-efficient tuning, and evaluate on broad benchmarks before and after tuning — not only on the target task.

Continued pre-training

Self-supervised training on a large unlabeled domain corpus (contracts, clinical notes, logs) to teach the model the domain's vocabulary and concepts. It reaches deeper than fine-tuning — it changes what the model knows, not just how it behaves — and demands far more data and compute. Amazon Bedrock offers continued pre-training for select models; the result, like fine-tuning, is a private custom model.

Model distillation

Transferring a large teacher model's capability on a task into a smaller, cheaper, faster student by training the student on the teacher's outputs. Amazon Bedrock Model Distillation automates the loop using your prompts to generate the teacher data. Expect near-teacher quality within the distilled scope and reduced general capability outside it — the right trade for high-volume, narrow workloads.

Confusion matrix

The grid of predicted vs actual classes whose four cells in the binary case — true positives, false positives, false negatives, true negatives — generate accuracy, precision, recall, and F1. Reading one is a reflex skill on MLA-C02: expect to compute a metric from raw counts under exam time.

F1 score

F1 = 2·(precision·recall)/(precision+recall). Because the harmonic mean collapses toward the weaker of the two, a model cannot buy a good F1 with precision alone or recall alone — which is why it replaces accuracy as the headline metric on imbalanced classes, where accuracy is dominated by the majority class.

LLM-as-a-judge

Evaluation technique where a strong LLM scores another model's outputs against a rubric — fast and scalable where no single reference answer exists. Known biases: position (favoring the first answer shown), verbosity (favoring longer answers), and self-preference (favoring its own model family). Mitigate by swapping answer order, writing tight rubrics, and spot-checking judge scores against human labels. Amazon Bedrock evaluations offer it as a managed mode.

Shadow testing

Validating a challenger model against real production traffic with zero user exposure: a SageMaker shadow variant receives a copy of a configurable share of requests, its responses are logged for comparison but never returned to callers, and you compare the variants before promotion. Contrast with A/B testing via production variants, where real users are split between models by traffic weights.

Hyperband

A hyperparameter tuning strategy in SageMaker Automatic Model Tuning built for expensive deep-learning jobs. Hyperband is multi-fidelity: it evaluates many configurations on partial training runs, uses the objective metrics they emit to stop the underperforming ones early, and dynamically reallocates the freed budget to promising trials. Compared with Bayesian optimization (sequential, learns between full trials) and random or grid search (which never use results at all), Hyperband leans on the metric return leg hardest — the exam cue is a large neural-network tuning job where bad configurations reveal themselves in early epochs.

Real-time inference

The default SageMaker AI hosting option: a managed HTTPS endpoint backed by instances you choose, invoked synchronously via InvokeEndpoint. Supports payloads up to 25 MB and 60-second processing (8 minutes for streaming responses), production variants, auto scaling, and deployment guardrails. The trade: capacity exists — and bills — whether or not traffic arrives.

Batch transform

The batch half of the exam guide's named strategies: a transform job reads input from S3, runs the model across the dataset on instances provisioned for the job, writes results to S3, and shuts down. Fits GB-scale datasets, scheduled scoring, and preprocessing; nothing remains deployed between runs.

Serverless inference

The option for intermittent, unpredictable traffic: SageMaker AI allocates compute per request against a memory size you configure, scales to zero when idle, and bills only for processing time. The costs are a 4 MB / 60-second envelope and cold-start latency after idle periods — the exam tells are "spiky", "idle stretches", and "pay only for what you use".

Asynchronous inference

For individually large or slow requests that need not answer synchronously: requests queue, process for up to an hour with payloads up to 1 GB, land results in S3, and can notify on completion. Because of the queue, the backing instance count can scale to zero — the "large payload + near real-time + no idle cost" combination in stems.

Production variant

The unit an endpoint config declares: a model paired with hardware and a traffic weight. Several variants behind one endpoint split traffic by weight (e.g., 90/10 champion/challenger) with per-variant CloudWatch metrics; TargetVariant on InvokeEndpoint pins a request to one variant. Also where the instance type lives — the classic "which object holds the hardware" trap.

Multi-model endpoint (MME)

The consolidation pattern for fleets of lightly-used, same-framework models (per-tenant, per-city): a shared serving container loads artifacts from S3 on first invocation, keeps hot ones in memory, evicts least-recently-used under pressure. Callers name the artifact with TargetModel. The trade is first-hit latency on cold models.

Inference pipeline

A serial chain of containers on one endpoint (its multi-container sibling instead allows direct invocation of up to 15 different containers via TargetContainerHostname). Pipelines keep fitted preprocessing with the model, killing training/serving skew; the same chain also runs in batch transform.

Provisioned Throughput (Bedrock)

Reserved capacity for a Bedrock model, purchased in model units for a commitment term, replacing shared on-demand capacity for high, predictable workloads. Two exam facts: it is the answer to "consistent high throughput without throttling," and models fine-tuned inside Bedrock require it for every family except Amazon Nova (whose fine-tunes can alternatively use on-demand custom model deployment).

Bedrock Knowledge Bases

The managed retrieval pipeline: connect a data source (S3, web crawler, Confluence/SharePoint/Salesforce), choose chunking (fixed, hierarchical, semantic, none, custom) and an embeddings model, and Bedrock indexes into your vector store (OpenSearch Serverless default; Aurora pgvector, S3 Vectors, Neptune Analytics, or third-party). StartIngestionJob re-syncs; queries support top-k, metadata filters, hybrid search, and rerankers.

Amazon Bedrock AgentCore

The production home for framework-built agents (Strands, LangGraph, CrewAI, OpenAI Agents SDK) with any model. Services compose: Runtime hosts agents and tool servers with per-session isolation and MCP/A2A protocol support; Gateway turns existing APIs and Lambda functions into MCP tools; Memory, Identity (inbound/outbound auth), Code Interpreter, Browser, and Observability round out the set. New to the exam in C02.

Action group

How a managed Bedrock Agent gets capabilities: the schema tells the FM what operations exist and what parameters they take; the executor (usually Lambda, or return-of-control to the caller) performs them. Agents add knowledge bases for grounding, session state and memory for continuity, and ship via versions and aliases.

Reranker

Second-stage retrieval: fetch wide (say top-25), rescore with a reranker model, pass only the strongest chunks to the FM. In Bedrock it runs via the standalone Rerank API or inside Knowledge Base Retrieve/RetrieveAndGenerate calls. Fixes "right documents, wrong order" cheaply, and often cuts generation cost by shrinking the prompt.

Bedrock Prompt Management

Treats prompt text and inference configuration like code: build a prompt, compare variants across wordings and models, snapshot a version when satisfied, and have applications reference prompt ARN + version. Shipping a prompt change becomes repointing a version — testable in pipelines like any other artifact.

Deployment guardrails

The native machinery behind UpdateEndpoint: stand up the new (green) fleet beside the old (blue), shift traffic all at once, by canary slice, or in linear steps, watch pre-specified CloudWatch alarms through each baking period, and roll back automatically on a trip. Rolling deployments update instance batches instead, avoiding a doubled fleet.

Blue/green deployment

The strategy family where the previous version stays alive while the new one takes traffic. Its superpower is instant, low-drama rollback; its cost is temporarily running two fleets. SageMaker deployment guardrails implement it with three traffic-shifting modes; the exam tell is "instant rollback" or "keep the old fleet until the new one proves out."

Canary deployment

The exposure-limiting traffic-shifting mode: a canary slice bakes on the new fleet under CloudWatch alarms; success promotes the remainder, an alarm rolls everything back having harmed only the slice. Linear shifting generalizes it to n equal steps. Exam tell: "minimize risk/blast radius while validating on real traffic."

SageMaker Model Registry

The system of record for deployable models: a model package group per problem, numbered packages per candidate carrying the inference spec (image + artifact), evaluation metrics, and lineage. Approval status (PendingManualApproval/Approved/Rejected) changes emit EventBridge events — the canonical hook for approval-gated deploy pipelines. Managed MLflow can auto-register models into it.

SageMaker Pipelines

The ML-specific orchestrator: define a DAG of steps (processing, training, tuning, condition, model registration), get step caching, retries, and lineage for free, and trigger runs from EventBridge schedules or drift events. Against the alternatives: Step Functions for workflows spanning many AWS services, MWAA for teams already invested in Airflow.

AWS CloudFormation

The provisioning engine underneath infrastructure as code on AWS: templates declare desired state, stacks create/update/delete resources as an atomic unit, change sets preview updates, drift detection exposes manual edits, and failures roll back. SageMaker resources (AWS::SageMaker::Model/EndpointConfig/Endpoint) deploy endpoint changes without downtime by swapping configs. The CDK synthesizes to it.

AWS CDK

The programmatic front end to CloudFormation: constructs (L1 raw, L2/L3 with sane defaults) bring loops, types, and reuse to infrastructure definitions, then synthesize into plain templates — stacks, drift detection, and rollback behave identically because it is CloudFormation underneath. Exam framing: CDK writes templates; CloudFormation executes them.

VPC endpoint (PrivateLink)

Interface endpoints place an ENI for a service (sagemaker.api, sagemaker.runtime, ECR, CloudWatch Logs) inside your subnets so API calls — including InvokeEndpoint — never cross the public internet; S3 uses a gateway endpoint via route tables. Distinct from VpcConfig (which routes the model containers' own traffic through your VPC) and EnableNetworkIsolation (which removes container egress entirely).

Data drift

Also called covariate shift: production inputs stop resembling the training data (new customer segments, inflation-moved amounts, upstream format changes) while the learned input→output relationship may still hold. Detected by scheduled statistical comparison of captured inference payloads against baseline statistics computed from the training dataset — no ground truth required, which is why it is usually the first drift signal available.

Concept drift

The world redefines what features mean: fraudsters adapt tactics, user intent shifts, policies change. Input distributions may pass every baseline check while accuracy quietly degrades. Confirmation requires ground truth (model quality monitoring), which arrives late — feature attribution drift serves as a label-free early proxy. The remedy is retraining on fresh labeled data.

SageMaker Model Monitor

Defines the exam-tested pipeline: enable data capture on an endpoint, baseline statistics/constraints from the training set, schedule monitoring jobs that compare captured traffic to the baseline, and emit violations as CloudWatch metrics for alarms and retrain triggers. Four monitor types: data quality, model quality (requires merged ground truth), bias drift (via Clarify), and feature attribution drift. As of late 2025 the service is in maintenance (closed to new customers) — the pattern and vocabulary remain in scope for MLA-C02.

CloudWatch generative AI observability

CloudWatch capability for observing generative AI applications: curated dashboards for Model Invocations (usage, token consumption, invocation logs) and Bedrock AgentCore agents, plus end-to-end prompt tracing across models, knowledge bases, guardrails, and tools. Integrates with Bedrock natively and with frameworks such as Strands Agents, LangChain, and LangGraph via the ADOT SDK.

Amazon Bedrock evaluations

Bedrock's evaluation capability: programmatic jobs with built-in metrics, judge-model jobs where a second LLM scores responses and explains each score, human-worker evaluations, and RAG evaluations that score retrieval and generation against ground truth. Run on a schedule over sampled production prompts, it becomes the in-production quality monitor for FM workloads — the text-model counterpart of accuracy tracking.

Random Cut Forest (RCF)

AWS's workhorse anomaly algorithm: an ensemble of random-cut trees assigns each point an anomaly score based on how much it changes the model of "normal". Streaming-friendly and label-free, it is the conceptual engine behind learned-band anomaly detection on metrics (CloudWatch anomaly detection) and is available as a SageMaker built-in algorithm for custom anomaly jobs over workflow and business metrics.

AWS Inferentia

AWS-designed accelerators dedicated to neural-network inference, exposed as Inf instance types for EC2 and SageMaker endpoints. Models are compiled through the AWS Neuron SDK. The exam signature: high-volume deep-learning inference where cost per inference matters and the team can adopt a compile step — versus general-purpose GPU (G-family) serving. Trainium is the training-side sibling.

SageMaker AI Savings Plans

A commitment-based pricing model for SageMaker AI usage: in exchange for a committed spend over a 1- or 3-year term, usage bills at reduced rates with no change to the workload itself. The exam tell is "steady and predictable for the foreseeable future". Commit after right-sizing — a Savings Plan on an oversized fleet locks the waste in. Spot remains the discount for interruption-tolerant training; the two solve different problems.

AWS Compute Optimizer

A recommendation service that studies CloudWatch utilization metrics and flags over- and under-provisioned compute, with suggested instance types and configurations. For the EC2/Lambda scaffolding of an ML platform it is the named right-sizing answer; for SageMaker endpoints specifically, Inference Recommender plays the equivalent role by load-testing candidate instance types against your model.

AWS X-Ray

X-Ray collects segments from each service a request touches and assembles them into traces and a service map, so cross-service latency and errors become attributable to a specific hop. In ML systems it answers "the prediction API is slow — is it the gateway, the Lambda, the feature lookup, or the model endpoint?" — the question per-service CloudWatch metrics cannot answer alone.

Bedrock AgentCore Observability

The observability layer of Amazon Bedrock AgentCore: built-in metrics for AgentCore primitives (runtime, memory, gateway, built-in tools, identity) plus session-level traces of agent decisions and tool calls, viewed on the CloudWatch generative AI observability dashboard's AgentCore tab. Agents hosted outside the AgentCore runtime join via ADOT SDK instrumentation. The exam answer for "trace what the agent did, per session, including tool failures and token usage".

AWS Cost Explorer

The analysis tool in Billing and Cost Management: visualize historical cost and usage, group and filter by dimensions including cost allocation tags (which must be applied to resources and activated in the billing console before they appear). For ML attribution, tag training jobs, endpoints, and pipelines by team/project, then filter. Complementary to AWS Budgets: Explorer explains spend that happened, Budgets alerts before and as it happens.

AWS Budgets

Budgets tracks cost or usage against limits you set and alerts on actual or forecasted breaches — forecasted alerts are the exam's "warned before the money is gone" answer. Budget actions go further than notification: apply a restrictive IAM or SCP policy or stop targeted EC2/RDS instances when a threshold is crossed. Pairs with Cost Explorer (analysis) and Service Quotas (hard resource ceilings).

Provisioned Throughput (Bedrock)

Bedrock's alternative to on-demand token pricing: purchase model units with commitment terms for dedicated, predictable throughput. Choose it when traffic is consistently high (on-demand throttling or token economics become the constraint) and when serving custom models fine-tuned in Bedrock, which deploy through Provisioned Throughput. It changes how capacity is bought — prompt-level token optimization still applies.

Execution role

Services act on your behalf: a training job assumes its execution role to read the dataset prefix and write its artifact path; an endpoint's role pulls its model and writes logs/metrics. Least privilege means one role per purpose, scoped to exact ARNs and prefixes — never a shared god-role. The caller creating the job must hold iam:PassRole for that specific role, which is the gate against escalating by attaching an over-privileged role to a job.

KMS customer managed key (CMK)

AWS KMS keys you create and govern (policy, rotation, disabling) rather than AWS-owned defaults. In ML pipelines they attach to the S3 locations for datasets and model artifacts (SSE-KMS), the ML storage volumes of training/processing jobs, and endpoint instance volumes. The key policy is a second, resource-side gate on access — and disabling the key revokes readability of everything under it, whatever S3 permissions say. Transit is covered separately by TLS.

AWS CloudTrail

CloudTrail records AWS API activity: management events (CreateTrainingJob, UpdateEndpoint, role changes) are captured by default, while data events — object-level S3 reads/writes such as "who downloaded the training dataset" — are higher-volume and must be explicitly enabled on the buckets that matter. The distinction is a reliable exam question. Pairs with AWS Config, which tracks whether resource configuration complies with rules rather than who called what.

Amazon Inspector

A vulnerability management service that automatically discovers workloads and scans them for known CVEs and unintended network exposure, producing prioritized findings. For ML delivery it gates CI/CD: the training and inference container images pushed to ECR get scanned before they ship their base-image CVEs to every endpoint. Out of scope in MLA-C01, in scope in C02. Distinct from Macie (sensitive data in S3) and CodeGuru Security (source-code flaws).

Amazon Bedrock Guardrails

Policy enforcement for generative applications, applied to inputs and outputs across models, agents, and knowledge bases. Six policy types: content filters (harm categories plus prompt-attack detection), denied topics, word filters, sensitive information filters (block or mask PII and custom regex), contextual grounding checks (filter responses unsupported by the source or irrelevant to the query), and automated reasoning checks. One versioned safety configuration replaces per-app prompt rules — and standalone evaluation lets the same policies cover models hosted outside Bedrock.

Shadow test

SageMaker AI shadow testing deploys a challenger as a shadow variant beside the production variant on the same endpoint: a configurable traffic sampling percentage of incoming requests is replicated to the shadow, while only the production variant's responses reach the calling application; shadow responses can be logged for offline comparison of latency and errors. One shadow variant per endpoint (alongside a single production variant); after reviewing the side-by-side dashboard you can promote the shadow to production. The contrast with an A/B test: shadows carry zero user risk but observe no user-facing outcomes.

SageMaker Inference Recommender

A SageMaker AI capability that benchmarks a model package across instance types (or serverless configurations) and recommends endpoint configurations with associated cost, throughput, and latency metrics — plus an initial instance count usable to seed an auto scaling policy. Default jobs produce quick instance recommendations; advanced jobs run custom load tests against your own traffic pattern and latency/throughput requirements. It is the endpoint counterpart of AWS Compute Optimizer, which covers EC2, Auto Scaling groups, EBS, and Lambda rather than SageMaker endpoints.

AIP-C01

The exam behind the AWS Certified Generative AI Developer – Professional credential. Announced with its beta on October 14, 2025; the standard version — refreshed to cover Amazon Bedrock AgentCore — has been open since mid-March 2026 (the beta closed March 31, 2026). Five content domains weighted 31/26/20/12/11, with Domain 1 (Foundation Model Integration, Data Management, and Compliance) the heaviest. Delivered at Pearson VUE or online proctored, in English, Japanese, Korean, and Simplified Chinese.

AIP-C01 exam guide (docs.aws.amazon.com), October 2026

Generative AI Lens (Well-Architected)

An AWS Well-Architected Framework lens for generative AI workloads built on Amazon Bedrock or customer-managed models on SageMaker AI. It walks the six pillars (operational excellence, security, reliability, performance efficiency, cost optimization, sustainability) across six lifecycle phases: scoping, model selection, customization, integration, deployment, and continuous improvement. The AIP-C01 guide names it (including the AWS WA Tool lens) as the mechanism for standardized, consistent GenAI implementations.

AWS Well-Architected Generative AI Lens (docs.aws.amazon.com)

Cross-Region inference (CRIS)

An Amazon Bedrock capability that routes model invocations across a set of AWS Regions through an inference profile. Geographic profiles (prefixed like us. or eu.) keep traffic inside a geography while Bedrock picks the destination Region by availability, latency, and demand; global profiles route anywhere the model is offered for maximum throughput. It raises aggregate throughput beyond one Region's quotas and absorbs regional capacity events — but it is a capacity mechanism, not disaster recovery, and inference profiles do not support provisioned throughput.

Amazon Bedrock user guide — cross-Region inference

Inference profile

The resource that powers Bedrock cross-Region inference: it names a foundation model and the set of AWS Regions that can serve requests for it. Applications invoke the profile ID exactly as they would a model ID (InvokeModel or Converse), which is why switching from a Region-pinned model to a resilient routed one requires no code change beyond the identifier.

LoRA (Low-Rank Adaptation)

A parameter-efficient fine-tuning technique: the base model's weights stay frozen while small low-rank matrices are trained and added to selected layers. The result is an adapter measured in megabytes rather than a full model copy — cheap to train, fast to deploy, easy to version per task, and swappable on a shared base model. That is what makes it the standard way to customize one base model for several product domains: many megabyte-scale adapters over a single copy of the base weights, each versioned and rolled back on its own.

PEFT (Parameter-Efficient Fine-Tuning)

The umbrella for fine-tuning methods that adapt a foundation model by training only a small set of added or selected parameters instead of all weights. LoRA is the best-known member. PEFT matters operationally: smaller artifacts to register and roll back, lower training bills, less catastrophic forgetting risk than full fine-tunes, and the ability to serve many task adapters from one base model.

Circuit breaker pattern

A resilience pattern borrowed from electrical engineering: when calls to a dependency keep failing, the breaker "opens" and requests fail fast to a fallback instead of queueing against a dead service; after a cooldown, trial requests probe ("half-open") before full traffic resumes ("closed"). The AIP-C01 guide names Step Functions circuit breaker patterns for resilient AI systems — typically a state machine checking breaker state in DynamoDB before invoking the model, with a graceful-degradation branch when open.

Graceful degradation

Designing a GenAI system so that failure of a component degrades the experience instead of ending it: fall back to a smaller or alternate model, serve cached or templated answers, drop to retrieval-only responses, or queue work for later. Paired with circuit breakers (which decide when to stop calling the primary) and cross-Region inference (which absorbs capacity events before degradation is needed).

AWS AppConfig

An AWS service for creating, validating, deploying, and rolling back application configuration and feature flags, separately from code. In GenAI architectures it holds the active model or inference-profile identifier (and prompt/parameter knobs), so switching providers or models is a configuration deployment rather than a code release. The usual shape is a thin invocation layer — Lambda behind API Gateway — reading the active model id from AppConfig: both the switch and the rollback are one configuration flip, with no redeploy and no per-provider branch to merge.

Converse API

The Amazon Bedrock runtime API that standardizes chat-style inference across foundation models: a messages array of alternating user/assistant turns, a separate system prompt, common inference parameters, document and image content blocks, and toolConfig for function calling. Contrast with InvokeModel, where the JSON body is model-specific per provider. Converse is the technical half of provider-switch architectures: change the model ID, keep the marshalling code.

Amazon Bedrock user guide — Converse API

Amazon Titan Text Embeddings V2

Amazon's second-generation text embedding model on Bedrock, tuned for retrieval. The output dimension is configurable — 1,024, 512, or 256 — which is the exam's favorite tradeoff: smaller vectors cost less to store and search faster, at some loss of representational fidelity. Supports floating-point and binary embeddings and an 8K-token input window. Pick the dimension by domain fit rather than by default, and measure the recall it costs against your own retrieval set first: changing it later means re-embedding the entire corpus, because vectors written by one configuration are not comparable with another's.

Amazon Bedrock model card — Titan Text Embeddings V2

OpenSearch Neural plugin

The OpenSearch capability behind "semantic search without writing your own embedding code": an ingest pipeline calls a connected ML model (commonly an embedding model on Amazon Bedrock) to vectorize documents as they are indexed, and the neural query type embeds the query text the same way at search time, running k-NN under the hood. Combine the neural leg with a lexical (BM25) leg in a hybrid query for the best of both. On OpenSearch Serverless, only remotely hosted models are supported.

Amazon OpenSearch Service developer guide — neural search

Hybrid search

Running a keyword query and a vector query over the same corpus and merging their normalized scores into one ranked candidate list. Lexical scoring (BM25) wins on part numbers, error codes, and names; vector similarity wins on paraphrase and meaning. Hybrid search catches both, and pairs naturally with a reranker that reorders the merged candidates. OpenSearch exposes it as the hybrid query type; Bedrock Knowledge Bases offers it as a retrieval configuration.

Hierarchical chunking

A chunking strategy that splits documents into large parent chunks and small child chunks. Children are what gets embedded and searched — small enough for precise semantic matching — but when a child matches, its parent is returned to the model, restoring the surrounding context a tiny fragment would lose. Amazon Bedrock Knowledge Bases supports it natively, alongside fixed-size, semantic, none, and custom Lambda chunking.

Amazon Bedrock user guide — chunking for knowledge bases

Semantic chunking

A chunking strategy that uses embeddings during ingestion: sentences (or sentence groups) are compared for similarity, and a new chunk begins where similarity drops — a topic boundary. The result is chunks that each carry one coherent idea, which embeds more faithfully than fixed-size slices that straddle topics. Costs more at ingest (it runs an embedding model) and shines on documents whose structure drifts across subjects.

pgvector

The open-source PostgreSQL extension for vector similarity search, supported on Amazon Aurora PostgreSQL (and RDS for PostgreSQL). It adds a vector column type, L2/cosine/inner-product distance operators, and IVFFlat and HNSW index methods. Its superpower in GenAI architectures is co-locating vectors with relational data: one SQL statement can join similarity results against business tables and metadata filters. A supported bring-your-own vector store for Bedrock Knowledge Bases.

Amazon Aurora user guide — pgvector

Amazon S3 Vectors

Object storage with vectors as a first-class type: vector buckets hold vector indexes you can query for similarity with sub-second performance, at a fraction (AWS cites up to ~90% storage-cost reduction) of a traditional vector database. The fit is large or cold corpora where cost dominates and query rates are modest; sustained high-QPS serving still belongs on OpenSearch. Integrates with Bedrock Knowledge Bases as a vector store option.

Amazon S3 user guide — S3 Vectors

HNSW (Hierarchical Navigable Small World)

The dominant approximate nearest-neighbor (ANN) index structure: vectors become nodes in a layered graph where upper layers make long hops and lower layers refine locally, so a query descends to its neighborhood in logarithmic-ish time instead of scanning everything. Higher recall and faster queries than IVFFlat at the cost of memory and slower index builds. Available in OpenSearch k-NN and pgvector — exam answers about "high-performance vector search at scale" usually sit on top of it.

Query decomposition

A query-handling technique for complex, multi-faceted questions: one embedding of a compound question is semantically diluted, so the question is split into sub-queries that each retrieve precisely, with the pooled chunks ranked together before generation. Amazon Bedrock Knowledge Bases ships it as the QUERY_DECOMPOSITION query transformation (RetrieveAndGenerate orchestrationConfiguration, or "break down queries" in the console); custom pipelines implement it with Lambda or Step Functions.

Amazon Bedrock user guide — query configuration

Query expansion

A retrieval technique that widens a query before search: add synonyms, acronym expansions, and related phrasings (an FM is good at generating them) so lexical and vector legs both find documents written in different words than the question. The complement of query decomposition: expansion widens one intent, decomposition separates several intents.

MCP (Model Context Protocol)

An open protocol that standardizes how AI applications and agents reach tools, data sources, and prompts: the application embeds an MCP client, each capability (a vector search, a document system, an internal wiki) is exposed by an MCP server, and every pairing speaks the same request/response contract. Exposing a vector query as an MCP tool is how several assistants built on different frameworks keep identical retrieval behaviour — one contract, versioned in one place, instead of a bespoke query path per application. On AWS, agent stacks such as Bedrock AgentCore use MCP to reach gateways of tools.

Function calling

The mechanism that turns a text generator into a system actor. The application advertises tool definitions (name, description, JSON schema) with the request — toolConfig in the Bedrock Converse API — and the model may respond with a toolUse block instead of prose. The application executes the call (for retrieval: a vector search), sends the toolResult back, and the model continues with real data in context. Retrieval-as-a-tool lets the model decide when to search, instead of always-on RAG.

Amazon Bedrock user guide — tool use (Converse API)

Amazon Bedrock Flows

The Bedrock feature for building GenAI workflows without application code: drag nodes — prompt, agent, knowledge base, Lambda, Lex, S3 retrieval/storage, condition — onto a canvas and wire them with data connections and conditional connections. Flows are testable in the console, then snapshotted as versions with aliases pointing at them, so the live workflow can be swapped or rolled back without touching the calling application. It launched in preview as "Prompt Flows", the name the AIP-C01 guide still uses for sequential prompt chains with branching and pre/post-processing.

Amazon Bedrock user guide — Flows

Chain-of-thought prompting

A prompt pattern that asks the model to work through intermediate reasoning before committing to an answer ("think step by step", or an explicit scratchpad section). It measurably improves multi-step and arithmetic-flavored tasks, at the cost of more output tokens and latency — which is exactly how exam stems frame it. In governed systems the reasoning is often kept in a structured section the application strips before display, so chain-of-thought and output-format specifications travel together.

Amazon Kendra

The managed intelligent-search product: built-in connectors (SharePoint, Confluence, Salesforce, S3, and dozens more — 40+ in all) crawl enterprise sources into one index, users query in natural language, and results are filtered by each user's source permissions — the ACLs a knowledge-base sync would flatten. The exam's elimination triangle: Kendra = permission-aware enterprise search; Bedrock Knowledge Bases = managed RAG retrieval where you own the generation; OpenSearch = you own the index and query DSL. Its GenAI index can also serve as a retriever behind Knowledge Bases and Amazon Q Business. Lifecycle note: Kendra closed to new AWS customers on July 30, 2026 (AWS points new builds at Knowledge Bases), but it remains on the AIP-C01 in-scope list.

Amazon Kendra developer guide (docs.aws.amazon.com/kendra/latest/dg/what-is-kendra.html)

Amazon Textract

The document on-ramp of multimodal pipelines: Textract reads scanned documents, PDFs, and images and returns not just the text (typed and handwritten) but the structure — tables as tables, forms as key-value pairs, plus targeted extraction via its Queries feature. In RAG over paper it runs first: scans become clean, structured text that validation, chunking, and embedding can actually use. Exam tell: inputs described as scanned PDFs, forms, or tables locked in page images mean Textract before anything else — a text embedding model has nothing to tokenize until it runs.

Amazon Textract developer guide (docs.aws.amazon.com/textract/latest/dg/what-is.html)

Amazon Rekognition

The image-understanding service of multimodal pipelines: label detection classifies objects and scenes, text detection reads printed and handwritten text in images, face APIs detect and compare faces, and content moderation flags explicit, violent, and otherwise unsafe imagery with granular labels. Two GenAI roles: at ingest, labels and extracted text become chunk metadata that makes images retrievable from text indexes; at request time, moderation gates user-supplied images before they join a multimodal FM call — a guardrail policy that inspects only prompt text never sees pixels.

Amazon Rekognition developer guide (docs.aws.amazon.com/rekognition/latest/dg/what-is.html)

Strands Agents

The agent framework the AIP-C01 guide names for building autonomous systems. Instead of a developer-authored flowchart, each loop pass hands the model the context plus tool descriptions and lets it natively reason, plan, and select tools; the SDK executes the selection and returns the result. Tools are standardized function definitions registered through the Strands API (decorated functions in Python). Scales to multi-agent via agents-as-tools and peer patterns, speaks MCP, and deploys anywhere — including AgentCore Runtime.

strandsagents.com documentation

AWS Agent Squad

The exam guide's named option for multi-agent systems. A classifier examines each incoming request together with conversation history and each agent's declared specialty, dispatches to the best-fit specialized agent, and maintains coherent context across agents and turns — heterogeneous specialists behind one front door. Available in Python and TypeScript.

AIP-C01 exam guide, task 2.1

ReAct (reason + act)

Thought → action → observation, looped until the task resolves. Model-driven frameworks run it implicitly; the exam guide highlights the explicit build: Step Functions states for each thought and tool call, giving audited execution history, bounded iterations, per-state timeouts, and configured retries — chain-of-thought with enterprise rails.

Human-in-the-loop (HITL)

The collaborative pattern: before irreversible or high-impact actions (refunds, deletions, external communications), the workflow pauses — Step Functions .waitForTaskToken — presents the proposed action for review, and resumes on the approval or rejection collected through an API Gateway feedback endpoint. Enforcement lives in the orchestrator, not the prompt.

Model ensemble

A model-coordination pattern: specialized FMs each attempt the task; aggregation merges — majority vote for classification, a judge model or scoring function for generations, or specialist-per-subtask with a composer. Trades multiplied inference cost for accuracy and robustness; the opposite economics of cascading and routing.

Model cascading

The exam guide's "API-based model cascading to perform routine queries": the default path is the task-tuned small model; a gate (confidence score, validation check, judge) passes most drafts straight back to the caller and escalates the rest. Savings scale with the non-escalation rate; the tax is added latency on escalated requests, which pay both models plus the gate.

Intelligent prompt routing (Bedrock)

The managed alternative to hand-built routing and cascading: invoke a prompt router like a model, and Bedrock analyzes each request, predicts response quality per candidate, and routes to the model offering the best quality/cost combination. Default routers cover the Amazon Nova and Meta Llama families; configured routers let you choose models, set the response-quality-difference criterion, and name a fallback model. The response reports which model actually served.

Amazon Bedrock user guide — prompt routing

GenAI gateway

The enterprise consumption architecture: every application calls the gateway instead of model providers; the gate enforces authentication/authorization, per-team quotas and rate limits, approved-model allowlists, and centralized guardrails, and logs each call for audit and cost attribution. Backends (Bedrock, SageMaker endpoints) swap without touching consumers; model credentials concentrate in one audited place.

AIP-C01 exam guide, task 2.3

Identity federation

The access half of enterprise integration: identity stays mastered in Entra ID/Okta/Ping; IAM Identity Center (workforce) or Amazon Cognito (customer-facing apps) exchanges assertions for short-lived AWS credentials; group-to-role mapping provides RBAC; least-privilege policies scope the roles to specific model ARNs and the two Bedrock invoke actions. Offboarding one directory account revokes everything.

AWS Outposts

Fully managed service extending AWS infrastructure, services, APIs, and tools to customer facilities — an Outpost is a pool of AWS compute/storage deployed at your site, operated and monitored by AWS as part of a Region. The exam's answer when data must be processed on-premises (jurisdiction, regulation, facility rules) while keeping the cloud programming model; pairs with Direct Connect/VPN secure routing for what may cross to the Region.

AWS Outposts user guide

AWS Wavelength

The edge-deployment option for latency-bound GenAI: application servers run inside the communications provider's network, so 5G device traffic terminates there instead of traversing the public internet — the ultra-low-latency lane for mobile AR/VR, gaming, and connected-vehicle GenAI features. Distinguish from Outposts (your premises, residency) and CloudFront (content caching, not your compute in carrier networks).

Large Model Inference (LMI) container

The concrete form of the guide's "container-based deployment patterns": specialized DLCs for deploying and optimizing LLMs on SageMaker AI, with the memory levers (quantization, tensor parallelism across GPUs), the utilization lever (continuous batching of token-generation steps), and token streaming built in. Sizing an LLM endpoint means memory for weights + KV cache, GPU utilization, and token-per-second capacity — not requests per second.

SageMaker AI developer guide — large model inference

Server-sent events (SSE)

The server holds an ordinary HTTP response open and emits incremental data: frames; browsers consume via EventSource, which reconnects automatically. Fits most GenAI UIs (display-only streaming) with no connection registry to operate. Carried end to end by Lambda response streaming (InvokeWithResponseStream) and API Gateway's streamed responses — provided no hop buffers.

WebSocket API (API Gateway)

API Gateway's stateful streaming lane: clients hold a two-way connection, and the backend pushes messages to a connection id. Earns its connection-management cost when the client talks mid-stream — cancel generation, typing signals, interactive multi-turn chat. For display-only streams, SSE is the lighter answer.

Exponential backoff

The baseline resilience control: each retry waits roughly twice as long as the last, randomized (jitter) so synchronized clients do not re-collide, with capped attempts. The AWS SDKs implement it natively via configurable retry modes. Absorbs transient 429s from quota-limited on-demand models; what backoff cannot absorb falls to fallbacks (secondary model, cross-Region, cached answers) — and sustained throttling signals a capacity decision, not a retry decision.

Amazon Bedrock Data Automation (BDA)

GenAI-powered transformation of unstructured content into structured formats through a unified inference API: intelligent document processing, media analysis, and modality-specific enrichment for RAG. Standard output extracts general information per modality; blueprints define custom output for documents and images; projects group output configurations. Built-in safeguards include visual grounding and confidence scores. The exam guide names it for automated data processing workflows.

Amazon Bedrock user guide — Bedrock Data Automation

Amazon Q Developer

AWS's generative-AI assistant for software development work: code generation and refactoring inside the IDE and the CLI, suggestions for unfamiliar API usage, and help writing tests for AI components and finding what is slow in them. It also serves as a troubleshooting aid, recognizing GenAI-specific error patterns alongside what CloudWatch Logs Insights and X-Ray report about the same request — the assistant reads the code, those two read the running system.

Amazon Bedrock Flows

A flow is nodes plus connections (data and conditional), built in the console's visual flow builder and invoked like any resource: input node → prompt/model/Knowledge Base/Lambda/condition nodes → output node. The exam guide's "no-code workflow builder" (named there as Prompt Flows), and one of the managed homes for prompt chaining.

Amazon Bedrock user guide — Flows

Prompt chaining

The guide's pattern for sophisticated applications: extract → analyze → draft → review as separate calls, so each step is individually promptable, testable, and re-runnable, and failures are attributable to a link rather than a monolith. Orchestrate with Step Functions (state-machine rigor), Bedrock Flows (visual), or agent frameworks (Strands, Agent Squad) when steps need autonomy.

SageMaker JumpStart

The low-effort lane to self-hosted foundation models: a hub of pretrained, publicly available models (plus solution templates) inside SageMaker AI that deploys a chosen FM — serving container, instance defaults, and configuration pre-packaged — onto a SageMaker endpoint from the Studio UI or a few SDK lines, with fine-tuning recipes for many entries. The middle answer between "the model is in the Bedrock catalog" (use Bedrock) and "we need container-level serving control" (build the endpoint yourself): you still own and pay for the endpoint, but skip the serving-stack engineering.

Amazon SageMaker AI developer guide — SageMaker JumpStart

AWS Secrets Manager

Where the non-AWS credentials in a GenAI stack belong: third-party model-provider API keys behind a gateway, MCP tool tokens, SaaS credentials for webhook and sync integrations. Secrets are stored encrypted (KMS), retrieved at runtime by the caller's IAM role, access-controlled with resource policies, and rotated on a schedule — replacing keys hardcoded in environment variables, prompt templates, container images, or code. The division of labor: IAM roles authorize SigV4-signed AWS calls; Secrets Manager carries every credential that is not one.

AWS Secrets Manager user guide

Defense in depth

The layers are complementary rather than redundant: each catches a failure class the others cannot see, so the design question is never “which control is best?” but “what gets through if this one fails?”. In AWS security that reads as edge, network, host, identity and data controls stacked over the same asset — a WAF rule, a security group, a hardened image, a least-privilege role and an encrypted bucket, any one of which can fail without exposing the data. In a generative AI application the same shape appears as pre-processing filters, model-level guardrails at the invocation, post-processing validation of schemas and tool allow-lists, and response filtering at the edge — a toxicity screen misses a politely-worded injection, and a guardrail knows nothing about your response contract. Exam tell: a question asking for defense in depth wants several layers, not the single strongest one, which is why it so often appears as a multiple-response item.

Prompt injection

Direct injection arrives in the user message ("ignore all previous instructions…"); indirect injection hides in content the application fetches — a web page, a PDF in the knowledge base, an email an agent reads. Defenses: the Guardrails prompt-attack filter on inputs, input tags so only untrusted turns are scanned, ApplyGuardrail over retrieved content before it enters the context window, least-privilege tools, and post-validation of model actions. Injection is a data-channel attack: anything the model reads is a potential instruction.

Jailbreak

Where prompt injection targets the application’s instruction hierarchy, a jailbreak targets the model’s alignment itself ("pretend you are an AI without rules…"). The Guardrails prompt-attack content-filter category detects both classes on the input side. Because new jailbreak patterns appear constantly, treat your corpus of known attacks as a regression suite: AWS updates the underlying filter models automatically, and your adversarial tests confirm behavior still meets your bar after every change.

Prompt-attack filter (PROMPT_ATTACK)

One of the six content-filter categories (hate, insults, sexual, violence, misconduct, prompt attack). It screens the input side — user prompts — for injection and jailbreak patterns, reporting confidence NONE/LOW/MEDIUM/HIGH against your configured filter strength. Pair it with input tags so system prompts and retrieved passages don’t false-positive, and with a defense-in-depth stack for whatever slips past.

ApplyGuardrail API

The request names a guardrail id and version, a source (INPUT or OUTPUT), and the content; the response returns action NONE or GUARDRAIL_INTERVENED, the masked or canned output, and per-policy assessments — including contextual-grounding GROUNDING/RELEVANCE scores against their thresholds. Because it is decoupled from foundation models, the same policy screens text from models hosted outside Amazon Bedrock, evaluates user input before an expensive retrieval step, and powers adversarial regression suites without paying for inference.

Guardrail input tags (selective evaluation)

For RAG and conversational apps, the full input often bundles system instructions, few-shot examples, conversation history, and retrieved search results around one new user message. Tagging lets the guardrail evaluate only the tagged section instead of re-screening trusted scaffolding on every turn — fewer false interventions and fewer guarded characters. Available through the AWS SDK, not the console or playground.

Contextual grounding check

The RAG hallucination control, enforced as policy: each model response is scored against the retrieved source passages (grounding) and against the user’s question (relevance), each with its own configurable threshold — raising a threshold tightens the gate. Scores surface in ApplyGuardrail assessments, so they double as confidence metrics you can log to CloudWatch and alarm on.

Automated Reasoning checks

You build an Automated Reasoning policy by uploading a source document (an HR guideline, an operations manual); the service extracts variables and rules, translates them to formal logic, and lets you review and edit them in natural language. At runtime the check mathematically proves whether a response contradicts the rules, highlights unstated assumptions, and cites the specific rules behind each verdict — a verification layer with structured feedback, unlike the binary gates of content filters and denied topics.

Structured outputs (JSON Schema)

Supported models accept a JSON Schema and are constrained to emit conforming JSON: on the Converse API, set outputConfig.textFormat.type to json_schema with strict mode; the portable fallback is defining a tool whose inputSchema is your schema and forcing the model to call it. Structure kills a whole failure class — prose wrappers, invented fields, missing keys — and is why AIP-C01 lists JSON Schema under hallucination reduction. It guarantees shape, not truth: values can still be wrong, so pair it with grounding.

Adversarial testing (red teaming)

The corpus grows from public jailbreak collections, your red team’s finds, and anything that ever escaped to production. Automated workflows (Step Functions on a schedule, or CI on every guardrail version, model swap, or prompt change) replay it through ApplyGuardrail — no model inference needed for input-side suites — and track two numbers: block rate on attacks and false-positive rate on a benign set. The exam guide names automated adversarial testing workflows explicitly.

Data masking

Comprehend’s character offsets make masking surgical: swap exactly the PII spans, keep everything else. Guardrails sensitive-information filters mask in flight (the ANONYMIZED action) in inputs and outputs. If a cleared workflow must restore identities, keep the placeholder↔value map encrypted and access-gated — separate from the masked text. Drop the map, and masking becomes anonymization.

Anonymization

Where masking preserves a re-identification path, anonymization destroys it: suppression, generalization, or placeholders with no map. The GenAI angle: anonymize fine-tuning corpora before training, because a customized model cannot selectively unlearn one person later — deletion requests against training data end in retraining without it. Choose masking when a cleared human needs the identity downstream; anonymization everywhere else.

S3 Lifecycle configuration

Expiration actions delete current versions after a set age; noncurrent-version expiration cleans history in versioned buckets; rules scope by prefix or object tags, so raw/ can expire at 90 days while redacted/ lives seven years in Glacier tiers. Pairs with Object Lock for the keep-this side of retention (legal holds) and CloudTrail data events for proof that deletion actually happened.

Model invocation logging

An account-level Bedrock setting, off by default: choose the modalities to capture (text, image, embedding, video) and the destination — S3 only, CloudWatch Logs only, or both. S3 delivery arrives as gzipped JSON batches queryable with Athena and cataloguable with Glue. The logs live in your bucket under your KMS key and your retention rules — the raw material for audits, drift analysis, and evaluation sampling, but also a PII store to govern like one.

SageMaker Model Cards

Model cards auto-populate training details for SageMaker-trained models and accept bring-your-own-info for models built elsewhere. Every edit creates a new immutable version — tamper-evident documentation — and the status field (draft → pending review → approved) integrates with EventBridge so pipelines can gate deployment on approval. Create and update them programmatically from the training pipeline: the exam guide asks for "programmatic model cards", and names them again for documenting FM limitations.

Data lineage

The Glue Data Catalog registers sources; data-source runs and Glue jobs emit OpenLineage-compatible lineage events; the SageMaker Unified Studio / DataZone catalog renders the graph with versioned history and column-level edges. For GenAI governance this is the "where did the data behind this answer come from" link in the evidence chain: knowledge-base corpora and fine-tuning datasets trace back to registered, tagged sources.

Bedrock agent tracing

Set enableTrace to true on an InvokeAgent request and each chunk of the response stream carries trace parts: PreProcessingTrace (how input was contextualized), OrchestrationTrace (the rationale for each step, action-group and knowledge-base invocations, observations), PostProcessingTrace (final shaping), FailureTrace (why a step failed), and GuardrailTrace (interventions). Persist traces to CloudWatch Logs or S3 for audit — and access-control them, because they can quote sensitive content. They are the raw material for user-facing reasoning displays.

Bias drift

Monitor fairness as named CloudWatch metrics — refusal-rate parity, outcome parity, tone scores per cohort — with alarms on divergence. In classic ML, SageMaker Clarify powers bias-drift monitoring against a launch baseline; for FM applications, scheduled LLM-as-a-judge evaluations over sampled production traffic play the same role per cohort. The response is the fairness A/B loop: revise the prompt or guardrail, test variants systematically, promote, and keep watching.

Prompt caching (Bedrock)

Amazon Bedrock feature for prompts that share a long static prefix (system rules, tool definitions, shared documents): cache checkpoints mark the contiguous prefix to reuse. Each checkpoint requires a per-model minimum of prefix tokens (currently 512–4,096), the TTL resets on every successful hit (many models: five minutes), and any edit upstream of a checkpoint is a miss — so variable content belongs after the static block. Cached reads bill at a discount and skip re-processing, but eligibility never guarantees a hit, and generation always runs: contrast Semantic caching, which skips the invocation entirely.

Semantic caching

A cache keyed on meaning rather than bytes: each query is embedded, compared against cached query vectors, and if the nearest neighbor clears a configured similarity threshold the stored response returns without invoking the model at all. Built on vector-capable stores such as Amazon ElastiCache (its documentation defines the pattern), it pays off wherever paraphrased duplicates dominate — helpdesks, FAQs, agent steps repeating similar lookups. Benefits: cost, latency, scalability within fixed quotas, answer consistency. Risks: a wrong match above threshold returns a confidently wrong answer, and stale entries outlive the truth — threshold tuning, tenant scoping, and invalidation tied to content updates are part of the design.

Edge caching

The shallowest cache layer: CloudFront stores responses at edge locations keyed on a deterministic hash of the request (method, path, normalized body — "result fingerprinting" in the exam guide's vocabulary) and serves exact repeats from the location nearest the user. The whole backend round trip — application, retrieval, model — is skipped. It only helps requests that repeat byte-identically, which makes it the fit for public, unauthenticated, shared content; the standing hazard is caching a personalized or private response into a shared key.

Latency-optimized inference

A per-request Amazon Bedrock option (in preview as of October 2026) that serves supported models on a faster tier without prompt, model, or architecture changes: performanceConfig: { "latency": "optimized" } on the runtime call. Verified roster at that date: Amazon Nova Pro, Anthropic Claude 3.5 Haiku, and Meta Llama 3.1 70B/405B, delivered through cross-Region inference profiles in US Regions. Once the optimized-tier quota is exhausted, requests are served at standard latency and billed at standard rates; the tier that actually served each request is visible in the API response and CloudTrail, with CloudWatch metrics split under the latency-optimized model ID.

Tiered model usage

The cost-capability pattern for GenAI traffic: most production queries do not need the most capable model, so a router grades complexity and dispatches accordingly. Implementations range from intelligent prompt routing (managed), to cascading (try the cheap model, escalate on low confidence), to distillation (train your own cheap tier from the flagship's outputs). The discipline that keeps tiering honest is measuring the tradeoff: per-tier quality on an evaluation set against per-tier cost — savings that arrive as quality regressions are not savings.

Batch inference (Bedrock)

Amazon Bedrock's offline lane: submit a job over JSONL records (InvokeModel or Converse format) in S3, and responses land in S3 when capacity allows, with EventBridge notifications on job state. AWS announced the pricing at 50% of on-demand for supported models — the discount you earn by not making anyone wait. Constraints that double as exam distractors: each record is one independent exchange (no tool calling, no structured output, no multi-turn), and batch is not supported for provisioned models. The natural pair is pre-computation: generate predictable answers overnight, serve them as lookups by day.

Prompt compression

The input-side half of token efficiency: audit what each prompt region costs (system rules, few-shot examples, conversation history, retrieved chunks) and cut what the outputs never use. Context pruning drops retrieval beyond what answers cite; history gets windowed or summarized; accreted system-prompt rules get consolidated. Two guardrails make it engineering instead of vandalism: compress against an evaluation set so fidelity loss is measured, and remember the complement — whatever static prefix survives compression is the ideal candidate for Prompt caching (Bedrock).

Time to first token (TTFT)

The perceived-latency half of LLM performance: everything before the first output token (network, queueing, retrieval, prefill of the whole input) is TTFT; everything after is generation. CloudWatch's TimeToFirstToken covers the ConverseStream and InvokeModelWithResponseStream APIs, while InvocationLatency measures to the last token. The derived diagnostic is output tokens per second — OutputTokenCount / (InvocationLatency − TimeToFirstToken) × 1000 as metric math — which separates "the model got slower" from "the answers got longer". Streaming UIs live and die on TTFT; prompt caching and shorter prefixes are its main levers.

Model invocation logging

Disabled by default and enabled in Bedrock settings: choose the modalities to capture (text, image, embedding, video) and the destination — S3, CloudWatch Logs, or both. Records carry invocation metadata with input and output bodies up to 100 KB inline; larger or binary payloads go to S3 only, under a data prefix. S3 delivery arrives as gzipped JSON batches queryable with Athena (catalogable via Glue, routable via EventBridge), which turns "what did we tell this customer last Tuesday" into a query. It is the only layer that holds the bodies themselves — CloudTrail records that an invocation happened and by whom, never what was asked or answered — and the exam's recurring trap is that nothing is captured retroactively: no enablement, no forensics.

Hallucination rate

No AWS service emits this number for free; you manufacture it. The production pattern: sample live responses (or route them through contextual grounding checks), score them on a schedule with judge-model or grounding evaluations against the retrieved sources, publish the failing share as a custom CloudWatch metric, and alarm on its trend alongside response-drift and token-anomaly signals. Deterministic complements: golden datasets re-run on every change and Output diffing against known-good baselines. Distinct from error rates — a hallucination is a successful request with wrong content, which is why availability monitoring never sees it.

Golden dataset

The GenAI equivalent of a regression suite: representative prompts (including known hallucination bait and edge cases) with vetted reference outputs, executed at near-deterministic sampling settings whenever the model, prompt template, retrieval configuration, or guardrail changes. Scoring combines exact or fuzzy matching, judge models, and Output diffing against the last known-good run. Because inputs never vary, any output change is attributable to the system change under test — the property live-traffic evaluation can never give you. The exam guide names golden datasets as the way to detect hallucinations and unique GenAI failure modes before users do.

Output diffing

Response-consistency analysis: hold inputs fixed, compare today's outputs against stored baselines — textually, structurally, or semantically via embedding similarity — and alarm when the difference distribution shifts. It catches what aggregate metrics hide: a model version update that quietly reformats answers, a retrieval change that drops a citation, a prompt edit that flips tone. Runs near-deterministic sampling so diffs reflect the system, not the dice; pairs with golden datasets in CI and with scheduled sampling in production, and with reasoning-path tracing to localize where a changed answer went wrong.

AWS Cost Anomaly Detection

A Billing and Cost Management feature that learns your spend patterns and alerts on deviations with a root-cause breakdown (service, account, tag), on a cadence faster than the monthly bill. In GenAI operations it is the money-side twin of CloudWatch anomaly detection on token metrics: CloudWatch sees the token burst within minutes; Cost Anomaly Detection confirms the dollar impact and catches spend CloudWatch metrics never see (vector storage growth, logging destinations). Complements AWS Budgets — thresholds you set — with anomalies you did not know to set thresholds for.

Continuous evaluation

The quality counterpart of monitoring: evaluation jobs run on a schedule (and on every change) against a golden dataset plus prompts sampled from production, with scores tracked over time so degradation shows up as a trend, not a surprise. On AWS this is Bedrock evaluation jobs triggered from pipelines or EventBridge schedules, results landing in S3 and dashboards. Continuous evaluation is what turns "regression testing for model outputs" and "automated quality gates" from ceremony into a system.

A/B testing (FMs)

Two configurations (model, prompt version, RAG settings) serve randomized shares of live traffic while the same metrics are collected for each arm: user feedback, judge scores on sampled responses, task success, latency, and cost per request. A/B answers "which is better for users"; canary answers "is the new one safe to roll out" — the exam distinguishes the intent. For FMs, pair the traffic split with LLM-as-a-judge scoring of sampled outputs, because click-through alone misses quality regressions.

Annotation workflow

The human half of user-centered evaluation: collect in-app feedback (ratings, thumbs, free-text reports), route flagged responses to reviewers, and record structured judgments. On AWS, Amazon A2I manages review loops over model outputs and SageMaker Ground Truth manages labeling workforces; the judgments become new golden-dataset entries, judge-rubric calibration data, and evidence for stakeholder reporting. Measure annotator agreement before trusting the labels.

Human evaluation (Bedrock)

One of the Bedrock evaluation modes: a work team compares or rates model responses against your custom prompt dataset using built-in or custom metrics (preference ranking, Likert ratings, thumbs). Use it where judgment outruns rubrics — brand voice, nuanced harm, domain correctness — and to spot-check that automated and LLM-as-a-judge scores still agree with people. Slower and costlier per item than automatic modes, so it samples rather than covers.

RAG evaluation

Bedrock evaluations can score a Knowledge Base or custom RAG source with LLM-computed metrics in two groups: retrieval quality (did the right chunks come back — context relevance and context coverage) and generation quality (is the answer correct, complete, and faithful to the retrieved context). Jobs take a dataset of queries with ground-truth texts/responses (some metrics run reference-free), normalize scores to 0–1 with per-score explanations, and compare across jobs — which is how chunking, embedding, and model changes are judged before rollout.

Faithfulness

A generation-quality metric in RAG evaluation: a judge model checks each claim in the response against the retrieved passages and scores how well the answer sticks to them. Low faithfulness with healthy retrieval scores means the model is embellishing beyond its sources — a generation-side fix (grounding instructions, guardrail grounding checks, model choice). Low faithfulness is routinely confused with wrong retrieval; the two-half structure of RAG evaluation exists to separate them.

Task completion rate

For agents, "sounded right" is not success: task completion rate scores whether the goal state was actually reached (ticket filed, refund issued, correct figure returned) across a task suite with checkable outcomes. It is measured by replaying a golden task set and verifying end states programmatically or with a judge, and it is the number stakeholders read first. Pair it with efficiency (steps, tokens, latency per completed task) or an agent can "win" by brute-forcing forty tool calls.

Tool-usage effectiveness

The step-level half of agent evaluation: correct tool selection for the sub-task, argument validity (schema-correct, right entities), call success rate, redundant or missing calls, and whether returned results actually shaped the next step. Computed from execution traces (AgentCore observability exposes tool-selection patterns and per-span timings), it catches agents that reach right answers through destructive or wasteful paths — the failure mode task-completion rate alone hides.

Semantic drift

A deployment-validation signal: re-run a fixed probe set on a schedule and after every model/prompt/KB update, embed the responses, and measure similarity against baselined answers. Falling similarity means the system now says different things to the same questions — a model version change, a prompt regression, or a shifted corpus — even when every response is individually fluent. String diffs cannot catch it because valid phrasings vary; semantic comparison can.

Context-window overflow

The classic GenAI content-handling failure: history, retrieved chunks, and instructions accumulate past the model maximum. Loud form: Bedrock rejects the call with ValidationException — "Input is too long for requested model." Quiet form: client code truncates before sending, and the model confidently answers from an amputated prompt. Diagnose by logging token counts per prompt segment and tracing assembled sizes; fix with dynamic chunking, history summarization, retrieval top-k budgets, and a hard token budget enforced before the call.

Embedding drift

Two distinct faults share the symptom of slowly worsening retrieval. Distributional drift: the corpus or user queries evolve (new products, new jargon) until stored vectors stop representing what people ask about — monitor retrieval scores over time and re-embed on a schedule. Model mismatch: documents indexed with one embedding model but queried with another (often after an upgrade) put queries and chunks in incompatible spaces — relevance collapses at once, and the only fix is re-indexing the full corpus with the new model.

Prompt observability

Prompt maintenance at scale needs prompts treated like requests: each invocation carries its prompt-version id, X-Ray segments record per-hop timing (retrieval, guardrail, model call), CloudWatch Logs keep the assembled prompt and response for diagnosis, and schema validation flags format inconsistencies in structured outputs. With that pipeline, "the assistant got worse Tuesday" becomes a query — which version, which segment, which token counts changed — instead of an archaeology project.

SCS-C03

The third revision of the AWS Certified Security – Specialty exam, in use from December 2, 2025. 65 questions, of which 50 affect your score and 15 are unidentified pilots; 170 minutes; results reported as a scaled score of 100–1,000 with a minimum passing score of 750; USD 300. Six content domains: Detection (16%), Incident Response (14%), Infrastructure Security (18%), Identity and Access Management (20%), Data Protection (18%), Security Foundations and Governance (14%). Scoring is compensatory — there is no per-domain pass bar. The target candidate has 3–5 years of experience securing cloud solutions.

AWS Certification exam guide (SCS-C03), October 2026

SCS-C02 (retired)

SCS-C02 had six domains: Threat Detection and Incident Response (14%), Security Logging and Monitoring (18%), Infrastructure Security (20%), Identity and Access Management (16%), Data Protection (18%), Management and Security Governance (14%). C03 split and restructured the first two into Detection and Incident Response, renamed domain 6 to Security Foundations and Governance, and moved weights — IAM rose from 16% to 20% and is now the heaviest domain. C03 also added content on OCSF edge integrations, generative-AI guardrails, inter-node encryption in transit, imported key material, sensitive-data masking, and multi-Region key and certificate management. Treat any course, practice set, or blog that lists "Security Logging and Monitoring" as a domain as pre-C03 material.

SCS-C03 exam guide, appendix comparing SCS-C02 and SCS-C03

Ordering item

New in SCS-C03 (with matching). The guide states the item "has a list of 3–5 responses to complete a specified task. You must select the correct responses and place the responses in the correct order to receive credit for the question." There is no partial credit for a nearly right sequence. Tactic: anchor the step that can only be first and the step that can only be last, then work inward on forced dependencies — containment before eradication, snapshot before termination, deny before investigate.

SCS-C03 exam guide, response types

Matching item

New in SCS-C03 (with ordering). The guide states the item "has a list of responses to match with a list of 3–7 prompts. You must match all the pairs correctly to receive credit for the question." Matching items are service-recognition items in disguise: the prompts are usually requirements and the responses are services or features. Tactic: commit the pairs you are certain about first, then let elimination resolve the rest — the last two prompts often decide each other.

SCS-C03 exam guide, response types

All-or-nothing grading

Multiple response is graded as an exact set; an ordering item needs every step selected and sequenced; a matching item needs every pair right. The consequences are strategic rather than philosophical. First, an unanswered question is scored as incorrect and there is no penalty for guessing, so a blank is strictly worse than a guess — never leave one. Second, a multi-part item is worth the same as a single-choice item, so a sequence you cannot reconstruct is worth a fast best guess and a flag, not eight minutes.

SCS-C03 exam guide, response types and exam results

Target candidate

The SCS-C03 guide describes a candidate with the equivalent of 3–5 years of experience securing cloud solutions, with working knowledge of the shared responsibility model, identity at scale, multi-account governance, software supply-chain risk, incident prevention and response, vulnerability management, layer 3–7 firewall rules at scale, root cause analysis, responding to an audit, logging and monitoring strategy, encryption in transit and at rest, and backup and disaster-recovery controls. It explicitly rules out designing cryptographic algorithms, packet-level traffic analysis, architecting whole cloud deployments, managing end-user compute, and training machine learning models. The out-of-scope list is a useful filter: if an option requires one of those skills, it is almost certainly a distractor.

SCS-C03 exam guide, target candidate description

In-scope service list

The guide publishes in-scope and out-of-scope service lists plus a technologies-and-concepts list, and warns that security affects all AWS services — a service can be out of scope overall while its security aspects are in scope (you will not be asked to configure S3 replication, but you will be asked about an S3 bucket policy). The in-scope list is long and deliberately broad: Athena, OpenSearch Service, SNS, Step Functions, API Gateway, EC2 (with Image Builder and Instance Connect), EKS, EMR, Lambda, Data Lifecycle Manager, Fault Injection Service, IoT Core, Bedrock, CodeGuru Security, Q Business, Q Developer, SageMaker AI, CloudFormation, CloudTrail (and CloudTrail Lake), CloudWatch, Config, Control Tower, Managed Grafana, Organizations, Resilience Hub, RAM, Service Catalog, Systems Manager, Trusted Advisor, User Notifications, Well-Architected Tool, Application Recovery Controller, VPC (Network Access Analyzer, NACLs, security groups, endpoints, Site-to-Site VPN, Flow Logs, Verified Access), Client VPN, CloudFront, Verified Permissions, Route 53 (with Resolver DNS Firewall), Direct Connect, ELB, Transit Gateway, Artifact, Audit Manager, ACM, CloudHSM, Cognito, Detective, Directory Service, Firewall Manager, Automated Forensics Orchestrator for Amazon EC2, GuardDuty, IAM, IAM Identity Center, Inspector, KMS, Macie, Network Firewall, Private Certificate Authority, Secrets Manager, Security Hub, Security Lake, Shield and Shield Advanced, STS, WAF, S3, Backup, DataSync, EFS, and FSx for Lustre. Only two services are named as out of scope: Amazon MWAA and AWS Payment Cryptography.

SCS-C03 exam guide, In-Scope AWS Services

Shared responsibility model

AWS is responsible for security of the cloud: hardware, the global infrastructure, the hypervisor, and the managed-service software. You are responsible for security in the cloud: your data and its classification, your identity and resource policies, your network exposure, your encryption choices, and — depending on the service — the guest operating system and application stack. Candidates rarely get the headline wrong; they get the boundary wrong. On EC2 you patch the guest OS; on Fargate you patch the container image but not the host; on Lambda AWS patches the runtime but you still choose when to move off a deprecated runtime version; on RDS AWS supplies patches but you choose and apply the maintenance window; on S3 there is no OS at all. One thing never moves: the correctness of your IAM and resource policies is always yours.

AWS shared responsibility model

Security “of” the cloud

Physical data-centre security, hardware and host decommissioning, the virtualization layer, the network backbone, and the code of managed services. You cannot audit it by inspection; you consume AWS’s audit evidence instead — SOC reports, ISO certificates, and PCI attestations downloaded from AWS Artifact. On the exam, any option that asks you to patch a hypervisor, inspect a physical facility, or harden a managed control plane is a distractor.

Security “in” the cloud

Everything you configure. The exam’s favourite consequence is blunt: AWS never misconfigures your bucket policy, never attaches an over-broad role to your instance profile, and never leaves your security group open to 0.0.0.0/0. When a scenario describes data exposure, the cause is on your side of the line by definition, and the right answer is a control you own — a resource policy, an SCP, a Macie or Access Analyzer finding, block public access, a KMS key policy.

Inherited controls

When an auditor asks how you control physical access to your servers, the answer is that the control is inherited from AWS and evidenced by AWS’s own audit reports, available on demand from AWS Artifact. Other controls are shared (patch management: AWS patches the infrastructure, you patch your guests) and the rest are yours alone. Knowing which bucket a control falls into is how you answer an audit question without claiming credit for work you did not do — and it is why Artifact, not a penetration test, is the right answer to “provide the auditor with evidence of data-centre controls”.

Preventive control

The AWS Well-Architected security pillar groups controls as preventative, detective, and responsive. Preventive controls remove the possibility: service control policies and resource control policies capping an organization, explicit deny statements, permission boundaries, S3 Block Public Access, mandatory encryption conditions in a bucket policy, a VPC with no internet gateway. The exam prefers them: a question that asks you to ensure something cannot happen wants prevention, and an answer that merely alerts on it afterwards is wrong however elegant.

Detective control

Detection does not block; it produces findings, and findings only matter if something consumes them. The exam tests the seams: GuardDuty analyses log sources without requiring you to enable those logs, Config records configuration state while CloudTrail records API calls, Security Hub aggregates everyone else’s findings rather than generating its own, Macie discovers sensitive data but does not redact it, and Inspector scans for CVEs rather than for behaviour. A requirement phrased “identify”, “discover”, “detect”, or “continuously assess” is asking for a detective control.

Responsive control

The automation that turns a finding into a change: an EventBridge rule on a GuardDuty finding invoking a Systems Manager Automation document that snapshots a volume and swaps the instance into an isolation security group, a Step Functions state machine orchestrating a documented runbook, Automated Forensics Orchestrator for Amazon EC2 collecting artefacts, AWS Backup restoring a clean copy. The exam’s preference ordering is prevent first, detect second, respond third — but it also prefers automated response over a human runbook whenever the stem mentions scale, consistency, or “least operational overhead”.

Least privilege

The default bias of every SCS-C03 answer. Operationally it means starting from no access and adding what is required, scoping resource ARNs instead of using a wildcard, adding conditions (aws:PrincipalOrgID, aws:SourceVpce, aws:RequestedRegion, tag matches) rather than widening actions, preferring short-lived role sessions over long-lived access keys, and using IAM Access Analyzer to generate a policy from observed CloudTrail activity instead of guessing. On the exam, between two options that both satisfy the functional requirement, the narrower one wins — and an option containing "Action": "*" or "Resource": "*" is almost never the key.

Explicit deny

IAM evaluation starts from an implicit deny, collects every applicable policy, and returns Deny if any of them says Deny. That makes an explicit deny the strongest tool you have and the most common reason a “correct” policy still fails: a deny in a service control policy, a permission boundary, a session policy, a resource policy, or a VPC endpoint policy will beat the allow you just wrote. Troubleshooting order therefore runs from the bluntest instrument down — organization policy, then boundary, then session, then identity and resource policies, then the key policy on any KMS key involved.

Blast radius

Measured in what a single compromised credential, instance, or key can touch. The boundaries that shrink it, strongest first: a separate AWS account (the hardest wall AWS offers), an organizational unit with its own policies, a VPC, an isolated subnet with no route to the internet, a security group, a role session with a short duration and a scoped session policy, and a per-workload KMS key whose key policy names only that workload. The SCS-C03 guide names minimizing the blast radius as part of preparing for incidents, which is why “separate accounts per environment” beats “separate IAM users per environment” whenever the stem talks about containment or isolation.

SCS-C03 exam guide, task 2.1

Attack surface

Reducing it is cheaper than defending it. The AWS levers the exam keeps reaching for: private subnets and VPC endpoints instead of internet paths, Session Manager instead of open SSH ports and bastions, hardened golden AMIs and minimal container images built by EC2 Image Builder, Inspector to find the packages you did not need, instance metadata service version 2 to close off server-side request forgery against the credential endpoint, and WAF plus Shield in front of anything genuinely public. Note the difference from blast radius: attack surface is how many doors exist, blast radius is how much lies behind the one that opens.

Separation of duties

On AWS this is usually implemented structurally rather than procedurally: a dedicated log-archive account that workload administrators cannot write to, a security-tooling account holding the delegated administrator for GuardDuty and Security Hub, organization-level CloudTrail that member-account admins cannot stop, S3 Object Lock so even an account administrator cannot shorten a retention period, and break-glass root access kept under multi-person control. When a stem says “even an administrator must not be able to…”, it is asking for a structural control — an SCP, an organization trail, Object Lock in compliance mode — not a stricter IAM policy in the same account.

WORM (write once, read many)

The immutability answer. S3 Object Lock protects object versions in two modes: governance mode, which a principal holding the bypass permission can override, and compliance mode, which no one can shorten or remove — not the bucket owner, not an administrator, not the account root user — until the retention period expires. S3 Glacier Vault Lock does the equivalent for a vault with a policy you lock permanently. A stem containing “regulatory retention”, “cannot be deleted even by root”, “ransomware-resilient backup”, or “tamper-proof audit evidence” is pointing here — and governance mode is the distractor when the requirement says nobody may override it.

Well-Architected security pillar

Organised around security foundations, identity and access management, detection, infrastructure protection, data protection, incident response, and application security — which is almost exactly the SCS-C03 domain list, and not by accident. The AWS Well-Architected Tool is the in-scope service that evaluates a workload against this guidance and produces an improvement plan, which is why it is the right answer to “evaluate our architecture against AWS security best practices” while Security Hub is the right answer to “evaluate our resources against security standards”.

AWS Well-Architected Framework — security pillar

Stem

A specialty stem runs two to six sentences: the setting (company, architecture, accounts), the event or need, any constraints, and then the question sentence carrying the qualifier. Security stems bury the requirement deliberately — a paragraph about a multi-account organization and a GuardDuty finding can turn on five words at the end, “without granting standing access”. Reading the question sentence first and then re-reading the body for the clauses that activate it is the single highest-yield habit on this exam.

Distractor

The recurring families: right service, wrong mode (batch analysis for a real-time containment need); technically correct but over-privileged (an administrator policy where a scoped role was asked for); correct but requires an application change when the stem said not to change the application; correct per account when the requirement was organization-wide; and correct but detective when the requirement was preventive. Recognising the family is faster than reasoning the option out from scratch, because the families repeat across all six domains.

Overlap trap

The signature SCS-C03 difficulty. The pairs that recur: Macie (what is in the data) against GuardDuty (what is happening to the account); Config (per-resource configuration state and remediation) against Security Hub (aggregated findings and standards across accounts); Inspector (known CVEs in images, instances, and functions) against GuardDuty (behaviour at runtime); a service control policy (caps what member accounts can do, never grants) against an IAM policy (grants within one account); Secrets Manager (managed rotation, cross-account resource policy, a price per secret) against Systems Manager Parameter Store (free standard tier, no built-in rotation); CloudTrail (who called which API) against Config (what the resource looked like); Detective (investigate and find root cause) against Security Hub (aggregate and prioritise). Learn the deciding clause for each pair rather than a ranking of the services.

Keyword reflex

Reflexes buy time, which is the scarce resource at 2.6 minutes a question. They are hypotheses, not answers: “detect crypto-mining” suggests GuardDuty, but if the stem adds “in container images before deployment” the answer moves to Inspector. The discipline is reflex, then check the qualifier and the constraints, then commit. Build the table from the exam guide’s own example services per task statement, because the guide’s phrasing is the phrasing the items use.

Two-pass strategy

With 65 questions in 170 minutes, the first pass should run faster than average — roughly two minutes an item — banking time for the second. Every flagged item still gets a committed best guess before you move on, because an unanswered question is scored as incorrect and there is no penalty for guessing. Ordering and matching items are the natural flag candidates: they read slowly and pay the same as a single-choice item, so a quick plausible arrangement plus a flag beats a long reconstruction attempt.

Pacing (2.6 minutes per question)

Useful checkpoints: question 20 by the 50-minute mark, question 40 by 100 minutes, all 65 touched by 150 minutes, leaving 20 minutes for flags. Two honest complications. Ordering and matching items take noticeably longer to read and arrange, so they consume the slack the easy items create. And you cannot tell which 15 of the 65 questions are unscored, so every question gets full effort regardless of how odd it looks.

Amazon GuardDuty

GuardDuty answers one question: is something malicious happening right now? It reads its foundational data sources directly from the AWS side and emits findings, each with a finding type and a severity value from 1.0 to 10.0 grouped into four levels — Critical (9.0–10.0), High (7.0–8.9), Medium (4.0–6.9) and Low (1.0–3.9). Across an organization you enable it once from a delegated administrator account and auto-enrol new accounts; noise is managed with suppression rules and trusted-IP/threat-IP lists. Keep its lane straight against its neighbours, because the exam tests the four-way split constantly: GuardDuty finds threats, Inspector finds vulnerabilities, Macie finds sensitive data, and AWS Config finds misconfiguration.

GuardDuty foundational data sources

This is the most reliably mis-answered fact about GuardDuty. It consumes these streams independently of your own logging configuration — you do not create a trail, a flow log or a resolver query log to feed it, you are not billed for those logs through GuardDuty, and turning your own trail or flow log off does not blind it. The practical consequence for a scenario question: "we disabled VPC Flow Logs to save money, will GuardDuty still detect this?" is yes for GuardDuty’s own analysis, while your Athena queries and dashboards go dark. If you need the records for investigation or evidence, you still enable and retain your own logs — GuardDuty reads a copy, it does not keep one for you.

GuardDuty protection plan

The foundational sources cover the control plane and the network. Protection plans cover everything else, and as of October 2026 they include S3 Protection, EKS Protection, RDS Protection, Lambda Protection, Malware Protection and Runtime Monitoring. Two exam-relevant properties: they are opt-in per plan (enabling GuardDuty does not enable all of them, so "we have GuardDuty" does not mean "we would have seen that"), and they are managed organization-wide from the delegated administrator account. When a question names a specific workload — an RDS login anomaly, a malicious file on an EBS volume, a suspicious process in a container — the answer usually turns on whether the matching plan is switched on.

GuardDuty Runtime Monitoring

Everything else in GuardDuty is read from AWS-side telemetry; Runtime Monitoring is the one plan that puts a security agent inside the workload, which is what lets it see process execution, file access and in-container network behaviour that no control-plane log records. The agent can be managed by GuardDuty or deployed yourself (an EKS add-on, an SSM-managed install). When runtime events arrive from an instance, GuardDuty stops charging for that instance’s VPC flow log analysis so you are not billed twice for the same visibility. Reach for it when the question describes detecting activity inside a running container or instance — crypto mining processes, suspicious shells, a reverse tunnel.

GuardDuty finding type

A finding type such as CryptoCurrency:EC2/BitcoinTool.B!DNS decomposes into the threat purpose (what stage of an attack: Recon, UnauthorizedAccess, Backdoor, CryptoCurrency, Impact, Discovery, Persistence, Policy, PrivilegeEscalation, Exfiltration, Trojan), the affected resource type (EC2, IAMUser, S3, Kubernetes, RDS, Lambda), and the family. You do not memorise the catalogue; you learn to read it. UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS says instance role credentials are being used from outside AWS — stolen metadata credentials, and the only fix that actually works is revoking the session, not blocking an IP.

GuardDuty suppression rule

A suppression rule combines filter criteria (finding type, resource id, account, tag) and archives every future match automatically. Suppressed findings are still generated and still retained; they are simply archived rather than delivered to EventBridge or Security Hub CSPM. Use it for the vulnerability scanner that trips Recon:EC2/Portscan every night, or for a bastion whose SSHBruteForce findings you already understand. Do not confuse it with disabling a protection plan (which stops the detection entirely) or with a trusted IP list (which stops the finding being raised at all).

Trusted IP list / threat IP list

A trusted IP list suppresses GuardDuty findings for traffic from those addresses before any finding exists — a corporate egress range, a penetration-testing provider. A threat IP list is your own intelligence feed: traffic to or from those addresses raises findings regardless of GuardDuty’s own view. Both are plain text files held in S3, both act on network findings rather than on IAM activity, and in an organization both are uploaded once by the delegated administrator and applied to every member account. The distinction the exam draws is against suppression rules, which archive findings after the fact instead of changing what is detected.

Delegated administrator

The multi-account pattern behind GuardDuty, Security Hub CSPM, Macie, AWS Config, Inspector, Detective, Security Lake and others: the management account designates one member — conventionally a dedicated security-tooling account — as that service’s delegated administrator, and the service is then configured centrally rather than account by account. Two properties matter on the exam. The designated account gains visibility into findings across the organization and can turn features on in members. And auto-enable for new accounts closes the gap an exam question always probes: without it, every account created after rollout is a blind spot. AWS recommends keeping the Organizations management account out of this role.

AWS Security Hub (CSPM)

Security Hub does two jobs the exam keeps separate. It aggregates — integrated AWS services and partner products send findings to it, and cross-Region aggregation plus a delegated administrator collapse an entire organization into one view. And it assesses — enabling a security standard switches on that standard’s controls, which run continuous automated checks (most of them implemented as AWS Config rules) and report a control status and an overall security score. From there, findings route onward through EventBridge, and custom actions let an analyst fire a chosen EventBridge event from a finding in the console to trigger a ticket or a runbook. Naming note: as of October 2026 the posture service is branded AWS Security Hub CSPM, while “AWS Security Hub” also names a newer exposure-management experience built on top of it; the SCS-C03 guide says “AWS Security Hub”, and the behaviour it tests is the CSPM behaviour described here.

Security standard

Enabling a standard enables the controls that belong to it, and a control shared by several standards is checked once and billed once. AWS Foundational Security Best Practices (FSBP) is the AWS-curated general-purpose standard and the sensible default; CIS AWS Foundations Benchmark is the externally defined configuration benchmark; PCI DSS and NIST SP 800-53 Rev 5 exist so a compliance team can read posture in the language of its own framework. As of October 2026 the catalogue also includes an AWS Resource Tagging standard and an AI Security Best Practices standard. A standard is an assessment framework, not an enforcement mechanism: it tells you a resource deviates, it does not stop the deviation.

Security control (Security Hub)

Controls are the unit you actually operate: enable or disable them individually, and a disabled control stops generating findings and stops counting toward the score. Most are implemented with AWS Config rules behind the scenes, which is why the configuration recorder is a hard prerequisite — with recording off, controls sit at NO_DATA and the dashboard looks healthy because nothing is being evaluated, not because everything passes. That failure mode is a favourite exam scenario. Controls carry severity, remediation guidance, and a compliance status that rolls into the standard’s score.

Security score

Security Hub computes the score from control status, excluding controls you disabled and controls with no applicable resources. Two properties matter when a question dangles the score at you. It is relative to enabled controls, so disabling failing controls raises the score without improving anything — the reason score movements get reviewed alongside the list of disabled controls. And it says nothing about threats: a perfect score with an active UnauthorizedAccess GuardDuty finding is entirely possible, because posture and threat detection answer different questions.

Central configuration

Without central configuration, enabling Security Hub and choosing standards and controls is per-account, per-Region work that drifts the moment someone creates an account. With it, the delegated administrator authors configuration policies — which standards are on, which controls are enabled or disabled — and associates them with the organization root, specific OUs or individual accounts; accounts can also be left self-managed where a team genuinely needs its own settings. Policies apply across the home Region and all linked Regions, and newly created accounts inherit their OU’s policy, which is the gap the exam is usually testing.

Finding aggregation Region

Security Hub is a regional service, so without aggregation a multi-Region estate means opening the console once per Region. Designating an aggregation Region replicates findings, insights and control statuses from the linked Regions into it, with updates flowing back out so a finding resolved in the aggregation Region is resolved at source. Combine it with a delegated administrator and you get the exam’s canonical answer to “central visibility across accounts and Regions”: delegated admin collapses accounts, the aggregation Region collapses Regions. Findings remain in their originating Region as well — aggregation copies, it does not move.

AWS Config

Config answers questions no threat detector can: what did this security group look like on Tuesday, who changed it, which resources violate the encryption rule right now. The recorder captures a configuration item on every change; rules evaluate items and mark resources COMPLIANT or NON_COMPLIANT; conformance packs ship rules in bundles; aggregators collapse accounts and Regions into one view; and remediation actions wire a non-compliant verdict to an SSM Automation runbook. The exam’s four-way service split puts Config on configuration compliance — distinct from GuardDuty (threats), Inspector (vulnerabilities) and Security Hub (posture aggregation and standards).

Configuration item (CI)

A CI contains the resource’s attributes, its relationships to other resources, the related CloudTrail event id, and the time of capture. The timeline of CIs is what makes “show me this resource before the incident” a lookup rather than an investigation, and relationship data is what lets you walk from a public subnet to the instances in it. CIs are produced only while the configuration recorder is on, for the resource types you told it to record — so a recorder scoped to exclude a resource type leaves a hole that looks identical to compliance.

Configuration recorder

One recorder per account per Region, configured with which resource types to record (all, all including global, or a chosen list) and where to deliver snapshots and history. Turn it off and Config rules stop evaluating, Security Hub controls fall to NO_DATA, and the configuration timeline simply stops — all without an error anywhere. Deploy it org-wide through Control Tower or a CloudFormation StackSet rather than per account, and treat “is the recorder on, in this Region, for this resource type?” as the first question when compliance data is missing.

AWS Config rule

Managed rules cover the common checks (s3-bucket-server-side-encryption-enabled, restricted-ssh, cloudtrail-enabled) with parameters you set; custom rules run your own Lambda function or a CloudFormation Guard policy when the managed catalogue falls short. Change-triggered rules fire when a matching CI arrives; periodic rules run on an interval, which is how you catch resource types the recorder does not emit changes for. A rule produces a verdict, and optionally a remediation action — an SSM Automation runbook, run automatically or on approval, with a retry count — which is where detection turns into correction.

Conformance pack

A pack collects rules (managed and custom) plus their parameters and remediation into a single YAML artifact you deploy as an immutable unit, with a compliance score for the pack as a whole. AWS publishes sample packs mapped to frameworks — CIS, PCI DSS, NIST, HIPAA, operational best practices for a given service — and you can author your own. Deployed from the Config delegated administrator as an organization conformance pack, one action covers every account and future accounts, which is exactly the automation the exam means by “deploy AWS Config conformance packs” for regular assessment.

Config aggregator

An aggregator names source accounts (explicitly, or the whole organization) and source Regions, then surfaces their compliance results and resource inventory in the aggregator account — including advanced queries in SQL over the aggregated inventory, which is the fastest way to answer “every unencrypted volume in the company”. Aggregation is read-only: it does not enable recorders or rules in the source accounts, so an aggregator over accounts with recording switched off faithfully reports nothing. Pair it with an organization conformance pack, which does the enabling.

Systems Manager State Manager

Where Config detects drift, State Manager removes it. An association pairs an SSM document (install the CloudWatch agent, apply this agent configuration, ensure the SSM Agent is current, run this hardening script) with a target set — instance ids, a tag query, a resource group — and a schedule, and SSM re-applies it on that cadence whether or not anything changed. That cadence is the point: a manual fix lasts until the next AMI, an association survives it. It is the exam’s standard answer for “keep the CloudWatch agent installed and correctly configured across a fleet”.

Managed and custom data identifiers

Managed data identifiers are AWS-built detectors — credentials, financial, personal and health data across many countries — combining pattern matching with keyword and proximity heuristics to cut false positives. Custom data identifiers add a regular expression plus optional keywords, a maximum match distance and an ignore list, for formats like your own employee id or policy number. Both are selected per sensitive-data discovery job, and both produce findings that name the bucket, the object and the identifier matched. Macie reports; it never modifies or redacts the object — remediation is your pipeline’s job.

AWS User Notifications

User Notifications sits on EventBridge events and gives you notification configurations with delivery channels — email addresses, AWS Chatbot into Slack or Microsoft Teams, mobile push — plus a single console inbox where notifications from multiple accounts and Regions are readable in one place. For security work it is the human-facing layer: fine for routing a finding to an on-call mailbox or a channel. It is not the integration layer — machine consumers, fan-out to queues and automation targets still belong to EventBridge and Amazon SNS.

Management event (control plane)

Management events are the control-plane audit trail: who called which API, from where, with which credentials, and whether it succeeded. CloudTrail records them whether or not you create a trail — Event history keeps 90 days in every account at no charge — and the first copy of management events delivered by a trail carries no CloudTrail charge either, which is why “one organization trail” is almost always the right answer and “a second trail for the same events” is almost always the distractor. Read-only and write-only events can be filtered separately, a common noise-reduction move that also quietly deletes evidence if you exclude reads.

Data event (data plane)

Data events are how you learn which object was read, not merely that S3 was called. They are disabled by default on every trail, they are charged per event delivered, and at scale they dominate a logging bill — so the design move is selectors: log a specific bucket or prefix, writes only, or exclude a noisy data bucket. Two exam reflexes follow. “We cannot tell who downloaded the file” almost always means data events were never enabled, and the 90-day Event history will not contain them either. And GuardDuty S3 Protection consumes S3 data events from its own stream, so GuardDuty coverage is not evidence that you have them.

CloudTrail Insights event

Insights is a per-trail option that models normal API call volume and error rates and raises an Insights event when activity deviates: a sudden burst of DeleteBucket, an error spike after a credential rotation, a AccessDenied wave that signals enumeration. It needs no rules from you and it is charged separately per analysed event. Treat it as a complement to GuardDuty rather than a substitute — GuardDuty detects known threat behaviour against threat intelligence, while Insights detects change relative to your own history, including activity no threat feed would flag.

Organization trail

An organization trail applies to the whole organization: member accounts see it as read-only and cannot stop or delete it, new accounts are covered automatically, and the events land in one bucket owned by a central security account. It is the standard answer to “central, tamper-resistant audit logging across accounts”, and it closes the hole that per-account trails leave the moment someone creates an account. Make it multi-Region at the same time, because a single-Region trail misses activity in every Region you forgot about. Service-linked and global-service events are included.

AWS CloudTrail Lake

An event data store ingests CloudTrail management and data events, Config configuration items, and events from non-AWS sources, holds them for a chosen retention period measured in years, and exposes SQL queries directly — no bucket layout, no partitioning, no table definitions. You pay to ingest and to scan. Choose it when the requirement is “investigate historical API activity with SQL and minimal operational work”, and choose a trail to S3 plus Athena or Security Lake when you need the raw files, cheap long-term storage, or a pipeline that feeds other tools. A trail and an event data store are separate configurations; creating one does not create the other.

Log file integrity validation

With validation enabled, CloudTrail hashes each log file with SHA-256 and, every hour, writes a digest file listing the files delivered in that hour and their hashes. Each digest is signed with a CloudTrail-held private key and also carries the signature of the previous digest, forming a chain — so deleting a digest breaks the chain visibly. Digests go to a separate prefix in the same bucket, which lets you apply tighter permissions to them. aws cloudtrail validate-logs does the verification. This is the control an auditor asks for when the question is “prove this evidence is intact”; it detects tampering, it does not prevent it — that is S3 Object Lock’s job.

CloudWatch agent

Nothing inside an EC2 instance or on-premises server reaches CloudWatch Logs on its own: the unified CloudWatch agent reads the file paths you list and ships them to named log groups, and also publishes memory, disk and process metrics the hypervisor cannot see. Four things must all hold, and each is an exam scenario on its own: the agent is installed; it is running; its configuration file names the right paths, log group, log stream and Region; and the instance profile carries log-write permissions (CloudWatchAgentServerPolicy is the managed shortcut). Store the configuration in SSM Parameter Store and apply it with a State Manager association so a new instance inherits it instead of waiting for someone to remember.

Metric filter

A metric filter turns text into a number: define the pattern (a JSON selector such as { $.errorCode = "*UnauthorizedOperation" }, or a plain term match), name the metric it publishes, and then build a CloudWatch alarm on that metric. This is how the classic detective controls work — alarm on root account usage, on IAM policy changes, on console sign-in failures, on CloudTrail configuration changes. Two properties the exam leans on: filters apply only to logs ingested after the filter is created, so they are not retroactive; and the alarm, not the filter, is what notifies — a filter with no alarm is a metric nobody reads.

Subscription filter

Where a metric filter counts, a subscription filter forwards — a near-real-time stream of matching events to a destination for transformation, storage or search. Firehose to S3 is the cheap archive-and-query path; Lambda is the transform-and-enrich path; OpenSearch Service is the interactive-search path. With a destination in another account the stream crosses the account boundary, which is how a dedicated logging account receives every workload account’s logs. Filters are per log group, there is a limit on how many a log group can carry, and cross-account delivery needs both a destination policy in the receiver and the right role in the sender.

CloudWatch Logs data protection policy

A data protection policy names the data identifiers to look for (credentials, financial, personal and health types) and an audit destination; matching values are masked in the console and in API reads, and only principals with logs:Unmask see the raw value. It is a read-time control on data already ingested, which is the distinction that matters: it limits who sees the secret, it does not stop the application writing it. The durable fix stays upstream in the code. Masking belongs to Domain 5’s data-protection toolkit alongside Amazon SNS message data protection; it appears here because a logging architecture that hoovers up application logs will collect secrets eventually.

VPC Flow Logs

Create a flow log on a VPC, a subnet or a single interface; publish to CloudWatch Logs, S3 or Amazon Data Firehose; keep the default record format or define a custom format that adds the fields investigations actually need (pkt-srcaddr, pkt-dstaddr, tcp-flags, flow-direction, traffic-path). What they contain: who talked to whom, on what port, how much, and whether security groups and NACLs allowed it. What they do not contain: packet contents, DNS names, and traffic to the Amazon-provided DNS server, instance metadata at 169.254.169.254, the Amazon Time Sync Service, DHCP, or Windows license activation. A flow log’s record format and destination are immutable after creation — to change them you delete it and create a new one.

Transit gateway flow logs

VPC flow logs see a VPC’s own interfaces; a transit gateway flow log sees the hub, with fields naming the transit gateway, the attachment and the source and destination VPCs and accounts. That makes it the right instrument for east-west questions a per-VPC log cannot answer — which VPC is talking to which, what is traversing the Direct Connect or VPN attachment, whether segmentation between environments is actually holding. Enable it on the transit gateway or per attachment, publish to CloudWatch Logs, S3 or Firehose, and define a custom format exactly as you would for a VPC flow log.

Route 53 Resolver query logs

A Resolver query logging configuration attaches to one or more VPCs and records queries made through the Amazon-provided resolver: the name queried, the type, the response, and the instance that asked. It is the missing half of network visibility, because flow logs deliberately omit traffic to the Amazon DNS server — so a beacon tunnelling data out as long random subdomain labels is invisible in flow logs and obvious here. Destinations are CloudWatch Logs, S3 or Firehose. Pair it with Route 53 Resolver DNS Firewall when you need to block the domain rather than merely record the question.

S3 server access logging

Server access logs record requests to a bucket (requester, bucket, key, operation, response, bytes, referrer, user agent) and are delivered best-effort to a target bucket, typically within hours. CloudTrail data events cover the same requests with full identity context, in near real time, routable to EventBridge — at a per-event charge. The exam’s split: choose data events when you need prompt, identity-rich, automatable evidence; server access logs when you want inexpensive bulk request history and can live with delay and occasional loss. Logging a bucket to itself creates a delivery loop and is a configuration mistake worth recognising.

Amazon Security Lake

Security Lake collects from natively supported AWS sources (CloudTrail management and selected data events, VPC Flow Logs, Route 53 Resolver query logs, Security Hub CSPM findings, EKS audit logs, WAF logs and more) plus custom sources, runs ETL to convert everything to OCSF in Apache Parquet, and writes it to an S3 bucket per Region in your account — with Lake Formation tables over it and lifecycle-managed retention. A rollup Region consolidates several contributing Regions. You keep ownership of the data; subscribers — Athena, OpenSearch, QuickSight, a third-party SIEM — read it. It is storage and normalization, not detection: it raises no findings of its own.

Open Cybersecurity Schema Framework (OCSF)

OCSF is a collaborative open-source schema, with versioning rules and event classes, that AWS co-founded and Security Lake adopts. Its payoff is concrete: a source IP is the same field name whether the event came from CloudTrail, a flow log, a WAF log or a partner firewall, so correlation stops being a per-source parsing project and becomes a join. Security Lake converts natively supported AWS sources automatically; custom sources must arrive already in OCSF and Parquet, which is where the Lambda or Glue transform in your pipeline earns its keep. Newly emphasised in SCS-C03, and also named in the edge-integration task of Domain 3.

Security Lake subscriber

Query-access subscribers get a Lake Formation resource share and read the data in place with Athena, Redshift or OpenSearch — ideal for an internal analyst account. Data-access subscribers are notified of each new object (SQS or an HTTPS endpoint) and read the Parquet directly, which is how a third-party SIEM ingests continuously. You choose the sources and the accounts each subscriber may see, so one lake can serve a SOC that needs everything and an application team that may see only its own logs. The lake stays in your account throughout — subscribers are granted access rather than sent a copy.

CloudWatch Logs Insights

Logs Insights is the fastest path from “something happened recently” to an answer: query across multiple log groups at once, automatically discover fields in JSON events, aggregate with stats … by, and extract from unstructured lines with parse. It is bounded by what is in CloudWatch Logs and by its retention, and you pay per gigabyte scanned — so narrow the time range first, which is also the single most effective cost control. Reach for it for minutes-to-days investigations and for building an alarm-worthy query; reach for Athena or Security Lake when the window is months or the data lives in S3.

Amazon Athena

Athena needs no cluster: define a table over S3, write SQL, pay for the bytes the query reads. Two decisions dominate the bill and the latency, and the exam tests both. Partitioning by date (and account and Region) means a one-day query reads one day, not the whole bucket. Columnar Parquet instead of raw JSON or gzip means a three-column query reads three columns. Security Lake hands you both already. Athena is the right answer for months of history and ad-hoc SQL; it is the wrong answer for sub-second dashboards and for full-text search over free-form text.

Partition projection

Log data lands in .../region=.../year=2026/month=10/day=20/ forever, so the partition count grows without limit and MSCK REPAIR TABLE or a nightly ALTER TABLE ADD PARTITION job becomes its own maintenance burden. Partition projection declares the ranges — a date range, an enumerated set of accounts and Regions — in the table definition, and Athena derives the prefixes it needs from the query’s WHERE clause with no metastore lookup at all. For CloudTrail and flow logs it is the standard configuration: faster planning, no partition-management job, and queries that still prune to the exact days asked for.

Amazon OpenSearch Service

OpenSearch indexes events so an analyst can search free-form text and pivot interactively, with OpenSearch Dashboards for visualization and built-in alerting on query results. The trade is that indexed storage costs more than S3 and the index must be fed — a CloudWatch Logs subscription filter, Firehose, or a Lambda transformer — and sized and retained deliberately (hot, warm and cold tiers, index lifecycle policies). The pattern the exam rewards: OpenSearch for the recent window a SOC lives in, S3 with Athena or Security Lake for the long tail, and a lifecycle that moves data between them.

Amazon Managed Grafana

Managed Grafana is the visualization and correlation layer, not a store: you point a workspace at several data sources and build one dashboard that puts CloudWatch alarm state, an OpenSearch log panel and an Athena query side by side, which is exactly what “one screen for an incident” means in practice. Access is through IAM Identity Center or SAML with Grafana roles, and the workspace can reach data sources across accounts through a service role. On the exam it answers “visualize and correlate across sources” — while the normalization itself belongs to Lambda, Firehose or Security Lake.

Incident response lifecycle

The value of naming the phases is that it tells you which answer a scenario is asking for. Prepare is everything done before the alarm: roles that already exist, logs already flowing, runbooks already tested. Detect and analyse are where a finding becomes a confirmed incident with a known scope — validate before you act, because a finding is a hypothesis. Contain stops the bleeding without destroying what you need to understand it, which is why isolation usually beats termination. Eradicate removes the foothold, recover returns the service from a known-good state, and the final phase turns the whole thing into a change: a new control, a new detection, a corrected runbook. Two orderings inside this sequence are worth memorising because the exam leans on them: capture volatile evidence before you change a resource’s power state, and contain before you eradicate.

Runbook

A runbook answers "how do I do this?" — snapshot the volume, replace the security group, rotate the key — in a linear sequence with no decision criteria. Because it is deterministic it is the half of an incident response plan that becomes code: on AWS, a Systems Manager Automation document that can be launched from an OpsItem, triggered by an EventBridge rule, or started by an Incident Manager response plan, with each step’s output recorded against an execution id. Its failure mode is correct execution of the wrong response, which is why runbooks need playbooks above them.

Playbook

A playbook answers "what should we do, and who decides?" for a class of incident — credential compromise, data exposure, DDoS, ransomware. It names the decision criteria and severity thresholds, who holds containment authority, when legal and communications are engaged, and which runbooks each branch invokes. It cannot be fully automated because its value is in the choices, though Step Functions can carry some branching while human approval stays human. Its failure mode is confident decisions nobody knows how to execute.

Severity classification

Driven by three inputs: confidentiality impact (what data, whose), integrity and availability impact, and scope (one resource, one account, the organization). Regulatory exposure is a multiplier on the floor rather than an input. Each tier carries who is engaged and how fast, how often you communicate, and how much authority the responder already has — at the top tier a responder isolates production without asking, at the bottom they open a ticket. A detector’s own severity (GuardDuty HIGH, for example) is one input, not the answer: a MEDIUM finding against regulated production data can outrank a HIGH finding in a sandbox.

Systems Manager OpsCenter

OpsCenter closes the gap between a finding and the work: OpsItems can be created manually, from a CloudWatch alarm, from an EventBridge rule, or from a Security Hub finding, and associated Systems Manager Automation runbooks can be launched from the item with their output recorded against it. Pair it mentally with Incident Manager — OpsCenter is the work record; Incident Manager is the people and the clock.

OpsItem

The unit of work in OpsCenter, and the id you cite afterwards. An OpsItem carries the related resource ARNs so a responder does not have to re-find them, the investigation notes, and the record of which Automation runbooks were run against it and what they returned. Created by hand or automatically from alarms, EventBridge rules, and Security Hub findings. In an exam scenario, "one tracked item per issue with the remediation history attached" is the OpsItem signature.

Systems Manager Incident Manager

Where OpsCenter tracks the work, Incident Manager decides who gets woken up. Contacts have contact channels (SMS, voice, email) and their own engagement plans with built-in delays; escalation plans walk tiers until somebody acknowledges; response plans bind an impact level (1 critical through 5 low), a chat channel, the contacts to engage, and the runbooks to start — so one trigger creates the incident record, pages the right people, and launches the first runbook together. It also carries post-incident analysis against a template, which is the "learn" phase with a schema.

Incident-response role

The highest-value preparation in Content Domain 2, and the source of a reliable exam tell: during an incident you must not be inventing permissions, so any option that creates the responder’s role as a response step is wrong. Deployed centrally by CloudFormation StackSets so new accounts are never a gap, with a trust policy naming only the security account and an MFA condition. Permissions are response verbs — read logs and findings, describe resources, create and share snapshots, modify security groups and network ACLs, attach deny policies, revoke sessions — not administrator, and deliberately not write access to data stores. Rehearse it: access that has never been exercised is a guess.

Forensics account

Normally in the Security OU alongside the log archive. The properties that matter are all about an attacker who has already won inside the workload account: different credentials, separate KMS keys, a write-once evidence bucket, service control policies preventing deletion, and read-only cross-account roles into the environments it examines rather than write access. One prerequisite is decided months earlier — a snapshot encrypted with the AWS managed key for EBS cannot be shared cross-account at all, so forensic capture requires a customer managed key whose policy grants the forensics account use of it.

Break-glass credentials

For the case where the identity provider is down, the federation trust is broken, or the attacker holds the credentials you would normally respond with. Because a standing path to full privilege is exactly what an attacker wants, break-glass is defined by its controls: hardware MFA held by two named people, the secret split or sealed so one person cannot use it alone, a documented approval with a stated reason, a hard time limit, an alert on any use sent to people other than the user, and a scheduled test. Unmonitored, it is a backdoor; untested, it is a rumour. AWS Organizations centralised root access shrinks the problem by removing root credentials from member accounts entirely.

Shield Advanced proactive engagement

Requires three things in place beforehand: registered contacts with a verified phone number, Route 53 health checks associated with the protected resources so Shield knows what unhealthy means, and a Business or Enterprise Support plan for Shield Response Team access. Contact details alone are not enough — that gap is a recurring exam scenario. Remember the layer beneath it too: Shield Standard is automatic and always on; Shield Advanced is a paid subscription that protects nothing until someone enrols resources, one ARN at a time or organization-wide through AWS Firewall Manager, and DDoS cost protection applies only to enrolled resources.

Health-based detection (Shield Advanced)

It does two jobs at once: it sharpens detection during a real attack and it reduces false positives when your own traffic spikes legitimately — a launch day is heavy but healthy. It is also a prerequisite for proactive engagement, because the Shield Response Team needs a signal that a protected resource is actually suffering. Configure it per protected resource, not globally.

AWS Resilience Hub

Named by the exam guide for validating an incident response plan. Applications are assembled from CloudFormation stacks, Resource Groups, AppRegistry applications, Terraform state, or EKS clusters; the assessment statically compares the architecture with your declared recovery objectives, so optimistic plans fail quietly before the incident rather than loudly during it. The generated artefacts are the useful part: standard operating procedures as Systems Manager documents, alarms for the signals that would tell you recovery is needed, and Fault Injection Service experiment recommendations to verify the whole thing. Assess, generate the test, run the test, re-assess.

AWS Fault Injection Service (FIS)

An experiment template holds actions (stop or terminate instances, send an SSM command, disrupt network connectivity, inject API errors, interrupt power to an Availability Zone), targets selected by tag, id, or filter with selection modes, an IAM role FIS assumes, and stop conditions — the safety mechanism that makes production experiments defensible. A scenario library provides pre-built experiments for common failure modes. Know the boundary: FIS injects faults, not attackers. It will not generate a GuardDuty finding, so to exercise a detection-to-remediation chain you pair it with GuardDuty sample findings.

Game day

The layer that tests the system, as opposed to a tabletop exercise (which tests decisions, roles, and notifications by talking a scenario through) and automated continuous checks (which test that the preparation still exists). Each layer finds failures the others cannot: a tabletop will never find a StackSet that missed an account, and an automated check will never find that two teams each believe they hold containment authority. Measure four intervals and track them over time — time to detect, to engage, to contain, to recover — because each maps to a different fix. An exercise that produced no backlog items was a demonstration, not a test.

Tabletop exercise

Costs an hour of several people’s time and finds a failure class nothing technical can reach: unclear containment authority, no agreed regulatory clock, an escalation path whose owner does not know they own it. Its blind spot is everything technical, because participants describe what they believe happens. Run it quarterly and after any major change, and pair it with game days (which test the system) and continuous automated checks such as responder-role coverage, SSM Agent coverage, and log-delivery verification (which test that preparation has not quietly drifted).

Systems Manager Automation document (runbook)

The natural EventBridge target whenever the response happens on an instance — collect a process list, apply an isolation group, patch, reboot — because the SSM Agent polls outbound to Systems Manager endpoints, so the document still reaches a host you have just cut off the network. That property is why isolation and investigation can coexist. Documents carry approval steps, branching (aws:branch), rate control across many targets, and their own assumable role, and each execution records per-step output against an id you can cite in the incident file. Requires a managed instance with the agent running and an instance profile.

Amazon Application Recovery Controller (ARC)

Named by the exam guide among the services used to remediate incidents, and the only one that orchestrates recovery rather than remediation. Readiness checks continuously audit whether the other Region or cell is configured and scaled to receive traffic, so "we can fail over" is measured rather than hoped. Routing controls are on/off switches backed by a cluster whose data plane is deliberately independent of the Region you might be evacuating, with safety rules preventing unsafe combinations such as turning every cell off; Route 53 health checks follow the control. Zonal shift moves traffic away from an impaired Availability Zone in one call. When a scenario describes shifting traffic away from a compromised cell, hand-editing DNS is the wrong answer.

Remediation idempotency

Actions must converge on a desired state ("set the security group to sg-quarantine") rather than apply a relative change ("remove the newest rule"), because a duplicated delivery of a relative operation removes a second, legitimate rule. Idempotency is one of four guardrails that keep unattended remediation from amplifying an incident: a precise event pattern, an exclusion tag or OU every automation honours, idempotent actions, and a per-resource per-window circuit breaker that suppresses and alarms. Add a dead-letter queue and an alarm on target failures so the automation cannot fail silently for seven months.

Automated Forensics Orchestrator for Amazon EC2

Know what it is first: an AWS Solution — a reference implementation you install into your own accounts and then own, including the IAM, the forensics account, and the cost — not a managed service you enable with a checkbox. Its phases are triage, isolation, memory acquisition, disk acquisition, evidence handling, and investigation, with artefacts landing in a separate forensics account and a tracking store holding the chain of custody. The reason it wins is not cleverness but consistency: the ordering constraint that destroys evidence when violated is encoded in a state machine rather than remembered at 04:00.

Memory acquisition

The highest-volatility artefact and the one a responder under pressure skips. Memory holds injected code with no file on disk, secrets decrypted only in RAM, the live socket table, and caches — none of which survive a power-state change. Write the image to the evidence bucket rather than to the instance’s own disk, so you are not overwriting the volume you are about to snapshot. One preparation-only nuance: an instance launched with hibernation enabled writes RAM to its encrypted root volume when hibernated, which can preserve memory contents, but hibernation must be enabled at launch.

Order of volatility

AWS makes the order unusually crisp because the platform states what dies when. Memory is destroyed by any stop, reboot, or terminate. Instance store volumes are lost on stop and terminate and cannot be snapshotted — the only capture path is copying data off the running instance. EBS volumes survive a stop and are deleted on terminate when delete-on-termination is set, so you snapshot them while the instance runs. Host-local logs rotate, fill, and die with the volume. Centrally shipped logs (CloudTrail, VPC Flow Logs, CloudWatch Logs, Config) survive the host but not retention expiry, so you copy the window rather than rescue it. The immutable archive with Object Lock survives even a mistaken responder.

Chain of custody

Four habits produce it, all cheap at the time and impossible to retrofit. Write-once storage: the evidence bucket in the forensics account with S3 Object Lock, versioning on, deletion denied to everyone including responders. Hash at capture, so you can prove months later that the image you are analysing is the image you took. Record time in UTC with its zone always stated, never converted. Attribute every action — responders work through the named incident-response role with their own identity in the session, so CloudTrail answers "who took this snapshot?" without anyone having to remember. The test: could a different engineer, in six months, prove which instance this image came from and that nobody altered it?

Forensic artefact

Capturing a log or an image is not yet producing evidence, and the operative word is artefact: each one carries the incident id, the source resource ARN, the capture timestamp in UTC, the acquiring principal, and a hash. The artefact set varies by compute model — an instance gives you memory and volume snapshots plus host logs; an ECS task gives you the image digest, the task definition revision, and the already-shipped log stream; a Lambda invocation gives you the function version and code hash plus the log stream, which is why you capture the version and not "the function". Network and API activity are captured as the relevant log window together with the query that produced it.

Finding validation

A finding is a hypothesis: a detector is telling you that observed activity resembles a pattern associated with compromise. Validation reads the finding properly (type, severity, resource, principal, window), confirms the subjects exist and are what the finding says, pulls the surrounding CloudTrail to see what the principal actually called and from where, decides whether the behaviour is expected — verified with the owner, not assumed — classifies severity against your own table, and then scopes it: which other resources did this principal touch, which principals did it create, does the source address appear elsewhere. Skipping validation produces both failure modes: isolating production over a detector’s opinion, and closing a real compromise because it looked familiar.

Security incident (versus a finding)

The transition happens in the analyse phase and has three outputs: a verdict (real or not), a severity (yours, not the detector’s), and a blast radius that containment must cover. Everything downstream hangs off it — engagement, containment authority, the regulatory clock. Two exam reflexes follow. Handed a finding and asked for the first action, the answer is usually validate and scope, not isolate or rotate, unless the scenario has already confirmed compromise. Told a finding was a false positive, the follow-up is a precise suppression with a recorded reason and a review date, never disabling the finding type across the organization.

Containment

Climb from the network, through the identity, to the resource: isolation security group, network ACL deny, removing the egress route, detaching the instance profile, an explicit deny on the principal, session revocation by issue time, deactivating an access key, and finally stopping or terminating. Each rung trades blast radius and evidence for decisiveness. The exam’s default instinct: when a scenario says "contain while preserving the ability to investigate", the answer is isolate, not terminate, stop, or reboot — all of which destroy memory. The honest exception is active exfiltration with no stated forensic requirement, where stopping the bleeding wins.

Eradication

Four categories, none skippable. Persistence on the host (cron and systemd timers, modified authorized_keys, replaced binaries, kernel modules) and — the part teams forget — in the account: new IAM users, access keys, roles, modified role trust policies, new identity providers, Lambda functions and the EventBridge rules invoking them, modified launch templates and AMIs, new routes and endpoint services. Entry point: patch the vulnerability or fix the misconfiguration. Rotation: everything the attacker could have seen, not only what they used. Verification: scan with Amazon Inspector, diff against the known-good baseline, re-run the detection, and confirm the investigation’s indicators appear nowhere. A host rebuilt from a clean AMI into an account that still holds the attacker’s access key is not eradicated.

Isolation security group

The lowest containment rung and the default first action, because it is the only network control with no evidence cost — Systems Manager still reaches the instance through the agent’s outbound polling, so collection continues after containment. One mechanical caveat: security groups are stateful and track connections, so a rule change governs new connections and an already-tracked session can persist for a time. When seconds matter, add a stateless network ACL deny at the subnet, accepting that it contains every instance sharing that subnet.

Quarantine subnet

The honest version of a popular runbook step. You cannot relocate a running EC2 instance to another subnet: its subnet is determined by its primary network interface at launch. So quarantining a live host means an isolation security group plus a subnet-level network ACL deny plus removing the route to the NAT or internet gateway. A true quarantine subnet — no internet gateway, no NAT, interface endpoints only — is where you attach the restored forensic copy for analysis, so malware on the examined disk has nowhere to call.

Session revocation

Temporary credentials stay valid until they expire and no API invalidates them. Authorization, however, is evaluated against the role’s current policies on every request, so the way to stop a stolen session is to make its requests fail. A blanket deny on the role works and takes every other session using that role down with it; a deny conditioned on aws:TokenIssueTime denies only sessions issued before the cut-off, so legitimate workloads recover on their next credential refresh. That is exactly what the console’s "Revoke active sessions" writes as an inline policy on the role. Set the cut-off to just before the first confirmed malicious call, not to "now".

aws:TokenIssueTime

Used with a DateLessThan condition in a deny statement to revoke sessions issued before a chosen moment, which is the surgical containment lever for a compromised role. Its boundary condition is examined often: because the key appears only in requests signed with temporary credentials from STS, the technique does nothing against a leaked long-term IAM user access key — those are contained by setting the key to Inactive (preserving the record for investigation) and then deleting it. Related trap: detaching an instance profile stops future credential retrieval from the metadata service but leaves already-issued credentials valid until expiry.

Known-good material

You cannot prove a root-compromised operating system is clean, only that it is less obviously dirty, so recovery rebuilds rather than repairs: a new resource from signed, versioned images, with new credentials and a new instance profile, and data restored onto it from a recovery point predating the compromise. Establishing that date is investigation work — the earliest confirmed attacker activity from CloudTrail, Config, host logs and the Detective baseline, minus a margin — not a storage decision, because the most recent backup is the most likely to be infected. Verify the replacement before it serves traffic, and treat recovery as incomplete until root cause is addressed.

Amazon Detective

Ingestion is continuous and agentless, pulled from the services themselves. On top of the graph Detective builds baselines — what this principal normally calls, from where, at what volume, at what hour — which is what raw logs cannot give you. It answers "is this unusual for this principal", "what else is connected", "when did behaviour first deviate" (the timestamp recovery depends on), and "are these findings one story" via finding groups. Enabled per Region with an administrator account delegated through AWS Organizations, and it requires GuardDuty to have been enabled for at least 48 hours first — so it has no retrospective reach and belongs in the preparation bundle. Use the graph to scope and CloudTrail to prove.

Behaviour graph

Scope becomes a traversal rather than a search: walk from a finding to the principal, to every resource that principal touched, to the source address, to the other accounts where that address appears. Retention runs to roughly a year of aggregated history, which is why it answers questions a 90-day log search cannot, and the graph spans member accounts so a traversal crosses boundaries the way an attacker does. Its limit is the flip side of its strength — it is built for investigation, not for citable precision, so a report needing "at 14:03:22 UTC, principal X called Y from Z, request id R" comes from CloudTrail Lake or Athena.

Origin Access Control (OAC)

With OAC the distribution authenticates to the origin as the CloudFront service principal using SigV4, and the origin’s resource policy grants access only when aws:SourceArn matches that distribution. The S3 bucket keeps Block Public Access on and needs no website endpoint, no public ACLs and no presigned links. OAC supports S3, Lambda function URLs, MediaStore and MediaPackage v2 origins, and covers cases OAI never did — notably SSE-KMS encrypted objects, where the KMS key policy must also allow the distribution to decrypt. Origin access identity (OAI) is the legacy predecessor: S3-only, and correct in exam questions only as the thing being migrated from.

CloudFront signed URL

Your application signs the URL server-side with a private key whose public half sits in a CloudFront trusted key group; the distribution validates the signature and refuses anything expired or altered. A canned policy names one resource and an expiry; a custom policy adds a source IP range and an optional start time. Signed URLs suit a single file — a download link, an emailed export, an installer. They are not S3 presigned URLs: a presigned S3 URL bypasses CloudFront entirely, and with it the web ACL, the headers policy and the logs.

CloudFront signed cookie

Set once after your application decides entitlement, a signed cookie covers every object the policy’s resource pattern matches — a subscriber area, or a streaming session whose manifest pulls hundreds of segments. It always uses a custom policy, which is what makes wildcards possible, and it requires a client that stores cookies and a domain you control. Rule of thumb: one file and a fresh link means a signed URL; a whole library or a session means a signed cookie.

Response headers policy

Attached to a cache behaviour, it stamps security headers on the way out: Strict-Transport-Security, Content-Security-Policy, X-Content-Type-Options: nosniff, X-Frame-Options, Referrer-Policy. AWS ships managed policies (including a security-headers policy) and you can create custom ones — CSP being the header that is genuinely application-specific. Policies can also carry CORS configuration and remove headers you would rather not advertise. Because headers are a property of the response, this is the control for "every response must carry…" requirements, and WAF is the wrong answer to them.

Custom origin header (origin verification)

The pattern for non-S3 origins, where OAC does not apply: configure the distribution to send a header with a long random value, then add a WAF rule on the Application Load Balancer that blocks requests without it. Pair it with a security group limited to the com.amazonaws.global.cloudfront.origin-facing managed prefix list, and rotate the secret like any other credential. Without origin verification, an attacker who knows the origin’s address bypasses every edge control you configured.

S3 CORS configuration

A browser calling a different origin sends Origin and expects a matching Access-Control-Allow-Origin back; the bucket’s CORS configuration lists the allowed origins, methods and headers that produce it. Behind CloudFront the failure mode is distinctive: unless your cache policy or origin request policy includes the Origin header, CloudFront neither forwards it nor varies the cache on it, so the first cached response — carrying one site’s origin — is served to every other site. Symptom: CORS works for one caller and fails for the next, apparently at random.

Web ACL

Entries are evaluated from the lowest priority number upward and the first terminating action wins — Allow and Block terminate, Count never does, CAPTCHA and Challenge terminate unless the request carries a valid token. If nothing terminates, the web ACL’s default action applies. Each rule costs web ACL capacity units (WCUs) against a budget (5,000 by default, raisable). Associate a web ACL with a CloudFront distribution (Global scope, managed in us-east-1) or with regional resources: ALB, API Gateway REST APIs, AppSync APIs, Cognito user pools, App Runner services, Verified Access instances and Amplify apps. Not with Network Load Balancers — WAF needs an HTTP-aware resource.

WAF rule group

Rule groups let one ruleset be shared across web ACLs and managed centrally, including by AWS Firewall Manager. A reference statement places the group in your evaluation order, and you can override the whole group’s action to Count or override individual rules inside it — the surgical way to tune a managed group without discarding it. Marketplace groups are how third-party WAF rules enter a web ACL: the same reference mechanism, a vendor’s content, subscribed rather than authored — so the rules update under you and the override controls are the only tuning you own.

AWS Managed Rules

The catalogue covers broad OWASP-style protection (core rule set), known-exploited inputs, SQL injection, platform-specific groups (Linux, POSIX, Windows, PHP, WordPress), admin protection, the Amazon IP reputation and anonymous IP lists, Bot Control (common and targeted), Account Takeover Prevention for credential stuffing on sign-in paths, and Account Creation Fraud Prevention for fake sign-ups. AWS updates them underneath you. Deploy with the group’s action overridden to Count, measure false positives in the logs, then enforce — and use the labels the groups attach to write your own precise follow-up rules.

Rate-based rule

The limit ranges from 10 to two billion requests per window. The aggregation key is what makes the rule precise: source IP, forwarded IP (from X-Forwarded-For behind a proxy), or custom keys built from a header, cookie, query argument, URI path, HTTP method, label namespace or JA3 TLS fingerprint — and keys can be combined. Add a scope-down statement so the rule only counts requests to the expensive path. WAF re-estimates rates continuously but not instantaneously, so expect a short lag when a block starts and when it lifts. Against a distributed bot fleet that rotates addresses, a fingerprint key reaches what an IP key cannot.

Scope-down statement

Written as an ordinary WAF statement (URI path, header, geo match, IP set, and their logical combinations) attached to a rule group reference or a rate-based rule. Account Takeover Prevention inspects only /login; body-inspection rules skip static assets; a rate limit counts only /search. It is also the right way to exempt a trusted source from one specific rule — far better than a blanket Allow at the top of the web ACL, which exempts that source from everything.

WAF label

Managed rule groups label prolifically: which sub-rule fired, which bot category was detected, which signal was observed. Because labels are visible to rules later in the evaluation order and in the WAF logs, the professional pattern is to run a managed group in Count, read the labels, and then write custom rules that act on specific labels — Challenge the ambiguous bot categories, Allow verified crawlers, Block the self-declared scrapers. Labels turn a managed group from an on/off switch into a signal source.

OWASP Top 10

AWS Managed Rules cover the request-detectable classes well: injection, cross-site scripting, path traversal, known-bad inputs, malicious payloads. They cover broken access control, authentication flaws, insecure design and vulnerable dependencies not at all, because those are properties of the application rather than of a request. Say both halves in a design review: a web ACL is virtual patching and attack-surface reduction, and the code fix still has to happen. For generative-AI applications the relevant list is the separate OWASP Top 10 for LLM Applications.

OCSF ingestion

Amazon Security Lake converts its natively supported sources — CloudTrail management and S3/Lambda data events, EKS audit logs, Route 53 Resolver query logs, Security Hub CSPM findings, VPC Flow Logs and AWS WAFv2 logs — into OCSF events stored as Apache Parquet, partitioned per Region and account. Anything it does not natively collect (a vendor bot-defence feed, another CDN, an on-premises appliance) arrives as a custom source you map into OCSF yourself. AWS Verified Access access logs can also be emitted in OCSF. The value is that one schema lets a SIEM or an Athena query read AWS and non-AWS security data together.

AWS Shield Standard

There is nothing to enable, configure or pay for, and no per-attack console. It mitigates the common, most frequently occurring layer 3 and 4 attacks — SYN and UDP floods, reflection attacks — against AWS services, and combined with the capacity of the AWS edge it absorbs the great majority of volumetric attacks without anyone noticing. On the exam, an answer offering "enable Shield Standard" as a remediation is describing work that is already done.

AWS Shield Advanced

Protectable resources are CloudFront distributions, Route 53 hosted zones, Global Accelerator accelerators, Application, Network and Classic Load Balancers, and Elastic IP addresses. Subscribing alone protects nothing — protections are configured resource by resource, which is why AWS Firewall Manager is the organization-scale answer and why "we subscribed but the new load balancer was unprotected" is a process failure rather than a product one. Capabilities: resource-specific detection tuned to your traffic, automatic application-layer mitigation on protected web resources, health-based detection via a Route 53 health check, attack diagnostics and the global threat environment dashboard, Shield Response Team engagement, cost protection credits for scaling charges during a verified attack, and AWS WAF usage at no extra charge for protected resources. Priced as a monthly subscription per organization (about US$3,000 as of October 2026) plus data-transfer fees, on a 12-month commitment.

Shield Response Team (SRT)

With Shield Advanced and Business or Enterprise Support, you can engage the SRT during an attack; with proactive engagement configured, they contact you when a protected resource’s associated health check fails during a detected event. You can also pre-authorize the team to act on your AWS WAF web ACLs, which is what allows them to apply mitigation rules during an incident without waiting for your approval. Both capabilities depend on health checks being associated in advance — set up before the incident, not during it.

Golden AMI

Everything configured at boot can fail or drift; everything baked into the image is a property of every instance that launches from it, including the one Auto Scaling creates during an incident. A golden AMI carries the OS baseline (CIS or STIG hardened), the SSM and CloudWatch agents, and nothing else — notably no key pairs and no secrets. Because the AMI id identifies the version, "is this instance hardened?" becomes "which image did it boot?", which is a question with an auditable answer. Enforce it with launch templates, AWS Config rules on approved image ids, and SCP conditions on ec2:RunInstances.

EC2 Image Builder

A recipe is a parent image plus an ordered list of components (AWSTOE YAML documents — AWS supplies managed ones including STIG hardening and agent installation; you add your own) and a semantic version. An infrastructure configuration describes the ephemeral build instance, which Systems Manager drives, so the build instance needs an instance profile with SSM permissions. A distribution configuration copies the finished image to Regions and accounts, can re-encrypt snapshots with the destination’s KMS key, and can update launch templates. A pipeline runs on a schedule or on dependency updates, launches a test instance from the new image to run test components, and only distributes if the tests pass. Image lifecycle policies then deprecate and delete superseded versions, so old images do not linger as launchable options.

Immutable infrastructure

The alternative to in-place patching. A CVE is remediated by building a new image on a patched base, letting the pipeline test it, updating the launch template or task definition, and replacing instances or tasks. Drift becomes impossible by construction, rollback is a version pointer, and the evidence for an auditor is the image version plus its test results. It requires workloads that tolerate replacement and a build pipeline that exists; stateful servers, licensed software and anything you cannot casually replace still want Patch Manager. Both answers are correct on this exam — to different questions.

Instance profile

The role is the identity; the instance profile is the object that makes it attachable to an instance. The SDKs and CLI discover the credentials automatically from IMDS and refresh them, so no key material exists on disk and every API call is attributable to the role. Replacing static access keys on instances with an instance profile is the single most common remediation in this part of the exam. Pair it with IMDSv2 required, because the metadata endpoint that delivers those credentials is also the target of SSRF attacks.

Service role

Distinguished from an instance profile (an EC2 identity) and an execution role (the identity of code the platform runs for you) by who assumes it: the service itself. A service-linked role is the managed variant — created and maintained by the service, with a predefined policy you cannot casually widen and a name you will recognize by its aws-service-role/ path. When a question describes a service needing to touch your resources, the answer is a service role, and the trust policy is where the exam looks.

ECS task role

Declared per task definition and scoped to what that task actually does: these S3 prefixes, this DynamoDB table, this queue, this KMS key. When application code gets AccessDenied from an AWS API, the task role is where you look. Keep it separate from the task execution role: one is your code, the other is the infrastructure around it, and conflating them is how a container ends up able to read every secret in the account.

ECS task execution role

Its permissions are plumbing — ecr:GetAuthorizationToken and image pulls, logs:CreateLogStream and PutLogEvents, and secretsmanager:GetSecretValue or ssm:GetParameters for values injected through the task definition’s secrets block. Because it is used mostly before your code runs, its failures look like start-up failures: CannotPullContainerError, an unresolvable secret, or a task that runs with no logs at all. The task role, by contrast, fails at runtime with AccessDenied from an application API call. That symptom pair is the exam’s way of asking which role is wrong.

IMDSv2

Under IMDSv1 a plain GET to 169.254.169.254 returns the instance role’s temporary credentials, so any server-side request forgery in the application becomes credential theft. IMDSv2 requires a PUT to /latest/api/token carrying a TTL header, and the returned token must appear as a header on every subsequent request; the response also honours a hop limit (default 1) so the token cannot be relayed off the instance by a proxy or a bridge-mode container — raise it to 2 only if containers legitimately need metadata. Enforce rather than request: HttpTokens: required in launch templates, ModifyInstanceMetadataOptions for running instances, account-level IMDS defaults per Region, an SCP conditioned on ec2:MetadataHttpTokens, and an AWS Config rule to catch the rest. Disable the endpoint entirely for workloads that need no role.

Inspector scan mode (agent-based / hybrid)

In agent-based mode Inspector scans only instances the SSM Agent manages, giving continuous re-evaluation and deep inspection of application language packages. In hybrid mode it additionally scans eligible agentless instances by taking EBS snapshots and reading the inventory from them, which requires an EBS-backed instance with a supported filesystem (ext3, ext4, xfs) and re-scans roughly every 24 hours. Member accounts inherit the mode from the delegated administrator. When findings are missing for part of a fleet, this setting and Systems Manager coverage are the first two things to check — not scan frequency, because Inspector has no schedule to raise.

Inspector network reachability finding

Inspector evaluates the real path — security groups, network ACLs, route tables, gateways, load balancers — and reports open paths to an instance, then uses that exposure to adjust how urgent a vulnerability is. The same critical CVE scores differently on an internet-facing instance than on one in an isolated subnet, which is what turns a CVE list into a work queue. It answers "is there a path", not "is anyone using the path" — that second question belongs to VPC Flow Logs.

Patch baseline

Approval rules can carry an auto-approval delay in days (or an approve-until date), which is how "never on release day, always within ten days" becomes configuration rather than a policy document. Baselines also hold explicit approved and rejected patch lists for packages you must pin, a rejected-patches action, and a compliance severity. AWS provides default baselines per operating system; production usually runs a custom one. Remember the consequence: compliance is measured against your baseline, so an instance can be compliant and vulnerable at the same time — which is why Inspector’s independent view matters.

Patch group

Tag instances with the Patch Group key and register that value against a baseline; instances without the tag fall back to the default baseline for their operating system, which is rarely what anyone intended. One baseline applies per patch group per OS, so the pattern is a baseline per risk tier and a tag per fleet. Patch groups are also the unit maintenance windows target.

Maintenance window

Windows run tasks such as the AWS-RunPatchBaseline document with Operation set to Scan (report only) or Install (apply approved patches, rebooting if required). The cutoff stops new tasks starting near the end of the window; concurrency limits how many targets are worked at once; the error threshold stops the run after N failures, so a defective patch reaches five instances rather than five hundred. Most organizations scan continuously through a State Manager association and install on a window.

Session Manager

The SSM Agent maintains an outbound connection to Systems Manager, so administrative access needs no inbound security-group rule and no public address. Authorization is ssm:StartSession, ideally conditioned on instance tags, and every session start appears in CloudTrail. Session preferences set the KMS key for session encryption, the log destination (CloudWatch Logs or S3 — this is the transcript an auditor wants), idle timeout, maximum duration and the RunAs user. Port forwarding tunnels a local port to the instance or through it to a remote host such as a database, which keeps that database entirely private. Prerequisites, in the order they usually fail: the agent, an instance profile with AmazonSSMManagedInstanceCore, and — in a private subnet without NAT — interface VPC endpoints for ssm and ssmmessages (plus ec2messages for some features).

EC2 Instance Connect

ec2-instance-connect:SendSSHPublicKey places an ephemeral public key on the instance, valid for about a minute, and your client then connects over SSH as usual. It still needs an SSH network path, which historically meant a public address or a bastion; an EC2 Instance Connect Endpoint now provides that path privately, so a private instance can be reached over SSH without a public IP. Compared with Session Manager it keeps SSH and removes the permanent key, but it does not give you the session transcript, the shell preferences, or the no-agent/no-port combination. Reach for it when a requirement insists on SSH.

Amazon CodeGuru Security

The service SCS-C03 names for discovering and remediating vulnerabilities inside a pipeline, alongside Amazon Q Developer, which reviews code and proposes fixes in the IDE and on pull requests. CodeGuru Security runs as a CI step and returns findings with enough context to fix rather than merely to file. Distinguish it from the rest of the scanning family: Inspector scans built artifacts and running resources for known CVEs, Macie finds sensitive data in S3, Config evaluates resource configuration. A pipeline finding is only a control if the build can fail on it.

OWASP Top 10 for LLM Applications

A versioned list (current edition 2025) reframing application security for systems whose behaviour is generated rather than programmed, because instructions and data arrive in the same channel. The five classes a security engineer must answer on AWS: prompt injection (direct and indirect) — Bedrock Guardrails prompt-attack filtering, and running the guardrail over retrieved content before it enters the context window; sensitive information disclosure — PII filters on outputs, no secrets in prompts, authorization enforced in code; improper output handling — treat completions as untrusted input, validate against a schema, never execute or concatenate them; excessive agency — one least-privilege execution role per tool, with confirmation or human approval for high-impact actions; and supply-chain and poisoning risk — vetted models and adapters, and write access to the knowledge-base corpus restricted to a pipeline role. Add bounded consumption (quotas, token limits, rate-based rules) and model invocation logging.

Security group

Rules allow traffic by CIDR, prefix list, or another security group — the reference form that lets you write "the app tier may reach the database tier" without naming an address, so the rule survives instance replacement and re-subnetting. There is no rule order and no deny: if any rule in any attached group permits the traffic it is permitted, and everything else is denied implicitly. Because it is stateful, the response to an allowed connection needs no outbound rule. The default security group allows all traffic between its own members, which makes it a quiet lateral-movement path for anything launched without an explicit group.

Network ACL (NACL)

Applies to every interface in the subnet, which makes it a boundary an application team cannot edit away — and the only VPC control that can express an explicit deny. Rules are evaluated by number, lowest first, with an implicit deny at the end, so a deny must carry a lower number than a broader allow to have any effect (number in increments and leave gaps). Being stateless, each direction needs its own rule, and responses leave to the client’s ephemeral port, so NACL designs carry a 1024–65535 rule. The default NACL allows everything both ways; a newly created NACL denies everything until you add rules.

Ephemeral port

Ranges vary by client: Linux kernels typically use 32768–60999, Windows Server 49152–65535, and NAT gateways and Elastic Load Balancing use 1024–65535. Because a network ACL cannot know which client is calling, the practical recommendation is to allow 1024–65535 for return traffic in the relevant direction. If that feels uncomfortably coarse, the instinct is right, and it is the argument for putting real access control in security groups. The symptom of getting it wrong is diagnostic: the request arrives and is logged by the application, and the client hangs until it times out.

Stateful vs stateless filtering

On AWS, security groups and the Network Firewall stateful engine are stateful; network ACLs and the Network Firewall stateless engine are not. The distinction is the single most testable network fact on this exam because it produces asymmetric failures you can diagnose from the symptom alone: a blocked inbound packet means nothing arrives and nothing is logged on the host, while a blocked return packet means the application answers and the client hangs. The troubleshooting method follows from it — trace the packet in both directions and check the control that applies at each hop, including the route tables.

AWS Network Firewall

You create a firewall with a firewall policy and Network Firewall places a firewall endpoint in a dedicated subnet per Availability Zone; nothing is inspected until route tables send traffic to those endpoints, which is the deployment step teams forget. The policy holds stateless rule groups (5-tuple and TCP flags, evaluated by priority, with pass, drop or forward to stateful rules) and stateful rule groups (connection-aware 5-tuple rules, domain lists matched against the HTTP Host header and TLS SNI, and raw Suricata rules, with pass, drop, alert and reject). Default actions decide everything no rule decided — an egress allow-list is only an allow-list if the default is drop. Logging covers alert, flow and TLS logs to CloudWatch Logs, S3 or Firehose. It is the answer for egress filtering and east/west inspection; it is not an HTTP application firewall, which is AWS WAF.

Firewall endpoint

One endpoint per AZ, in its own subnet, with route tables redirecting the traffic you want inspected. Keep an endpoint in every AZ you inspect: routing inspection traffic across AZs costs money and couples your availability to another zone. The companion failure mode is flow asymmetry — if a connection’s two directions traverse different endpoints, the stateful engine never sees the whole conversation and the connection breaks; through a Transit Gateway, appliance mode on the inspection VPC attachment prevents it.

Suricata-compatible rule

Stateful rules are provided to the engine as Suricata strings, which is why community and vendor rulesets can be used directly and why evaluation order follows Suricata conventions. The policy chooses between default action order — pass rules considered before drop, drop before alert — and strict order, where rule groups run by priority and rules in the order you wrote them, with selectable default actions (drop all, drop established, alert all, alert established). The rule group’s own order setting must match the policy’s. Run new rules with alert before drop: the same measure-then-enforce discipline as WAF Count mode.

Inspection VPC

Transit Gateway route tables send spoke-to-spoke and internet-bound traffic to the inspection VPC instead of letting it flow directly, so one policy, one ruleset and one log stream cover every account. Two requirements make it work: a firewall endpoint in each Availability Zone you inspect, and appliance mode on the inspection VPC attachment so both directions of a flow stay in the same AZ. Without appliance mode you get the classic symptom — long-lived connections failing intermittently while short requests mostly succeed.

East/west traffic (and north/south)

Most environments are well defended north/south (CloudFront, WAF, Shield, TLS) and wide open east/west, which is what lets one compromised host become an incident. East/west controls: per-tier security groups referencing each other rather than CIDRs, isolated subnets with no internet-gateway or NAT route, Transit Gateway route-table isolation so spokes cannot reach each other, separate VPCs or accounts for genuinely separate trust boundaries, and an inspection VPC when the traffic between segments must be examined rather than merely permitted. Read the requirement literally: "protect the application from internet attack" is north/south, "prevent lateral movement" is east/west.

Transit Gateway route table

Give every spoke attachment a route table containing only the shared-services and inspection attachments, and spokes have no route to each other — no security group, NACL or firewall rule required, because there is no path. That scales: the twenty-sixth spoke changes no rules anywhere. Related facts the exam likes: VPC peering is not transitive (A↔B and B↔C never yields A↔C) and does not allow edge-to-edge routing through a peer’s NAT gateway, VPN or Direct Connect connection — both limitations worth reading as segmentation guarantees.

Network Access Analyzer

A scope declares what you want to be untrue ("no path from an internet gateway to these subnets"); the analysis evaluates route tables, security groups, NACLs, gateways, peering connections, transit gateway attachments, endpoints and load balancers, and each finding lists every hop in the path. A run with no findings is a provable negative about the configuration, which is why this belongs on a schedule and in the pipeline rather than in an annual review. It answers "is there a path"; VPC Flow Logs answer "is anyone using it", and you need both before you close a rule.

AWS Site-to-Site VPN

Each VPN connection provides two tunnels terminating on endpoints in different Availability Zones, because AWS performs maintenance on them; a design that configures only one will have an outage. Routing is static or BGP, and BGP is the default answer for anything that must fail over without a human — it also enables ECMP across tunnels on a transit gateway for throughput beyond a single tunnel’s roughly 1.25 Gbps. Encryption in transit is intrinsic, which is why a VPN is also the way to add encryption to a Direct Connect connection, and the same mechanics serve multi-cloud links to another provider’s gateway.

AWS Direct Connect

A dedicated connection (1, 10, 100 or 400 Gbps) terminates on a port you own; a hosted connection comes from a partner at smaller capacities. Virtual interfaces carve it up: a private VIF reaches a VPC through a virtual private gateway or Direct Connect gateway, a transit VIF reaches a transit gateway through a Direct Connect gateway (the way to serve many VPCs from one interface), and a public VIF reaches AWS public service endpoints. The exam’s favourite misconception is that a private circuit is an encrypted one: it is not. Add MACsec on a supported dedicated port, or run a Site-to-Site VPN over the connection. SCS-C03 removed cross-Region VIF design from the skill list.

MACsec (MAC Security)

You request a MACsec-capable port, create a CKN/CAK key pair (AWS stores it in Secrets Manager when you associate it), and configure the same key on your own MACsec-capable router. Only 256-bit keys are accepted; 100 and 400 Gbps connections require extended packet numbering, because the original 32-bit numbering space would force key rotation every few minutes. The limitation to state out loud: MACsec protects the link between your router and the AWS Direct Connect device — not the whole path to your VPC — and it is unavailable on hosted connections. "Encrypted end to end from on-premises to the application" is IPsec, or TLS, or both.

AWS Client VPN

Clients receive an address from the endpoint’s client CIDR block (which must not overlap any routed network and cannot be changed later) and then reach whatever the authorization rules grant — per-destination-CIDR, optionally scoped to an AD or SAML group. Associating a subnet creates the path; without an authorization rule, connected users reach nothing, which is the usual cause of "connected but no access". Split tunnel sends only endpoint routes through the tunnel; full tunnel sends everything. Security groups apply to traffic leaving the endpoint, connection logs go to CloudWatch Logs, and a client connect handler Lambda can add your own posture checks at connection time. Contrast with Verified Access, which grants one application rather than a network.

AWS Verified Access

A Verified Access instance holds trust providers — an identity provider (IAM Identity Center or OIDC) and optionally device providers such as Jamf, CrowdStrike or JumpCloud — and endpoints front applications behind an Application Load Balancer, a Network Load Balancer or a network interface. Each request is authorized by a Cedar policy reading the trust context (context.identity.groups, a device risk level), so a device that falls out of compliance is denied on its next request: there is no session to outlive the decision. Access logs carry the decision and its context and can be emitted in OCSF, and an AWS WAF web ACL can be associated with the instance. It is also an answer to "unnecessary network access" — the access you never grant needs no analyzer to find.

Route 53 Resolver DNS Firewall

Rule groups contain rules referencing your own domain lists or AWS managed domain lists for malware and command-and-control domains; rules are evaluated by priority and rule groups are associated with VPCs. ALERT permits and logs, which is the tuning posture before you block anything. A BLOCK returns NODATA (success with no record — the default, which looks like a timeout), NXDOMAIN, or OVERRIDE with a CNAME to a sinkhole you monitor, so malware connects to infrastructure you watch. DNS Firewall Advanced rules additionally score the shape of a domain name to catch DNS tunnelling and algorithmically generated domains at a confidence threshold you choose (LOW detects most and needs tuning; HIGH only well-corroborated threats). Resources that bypass the VPC resolver are not filtered, so pair it with egress controls on port 53 and on DNS-over-HTTPS.

Resolver query logging

The only data source that ties a domain to the instance that asked for it, which makes it the detection half of DNS security: exfiltration encoded in query labels, and command-and-control beaconing that shows up as a metronome. VPC Flow Logs cannot answer it, because by the time a connection is made the name is gone. GuardDuty separately analyses VPC DNS activity and raises command-and-control and crypto-mining findings without query logging being enabled, so run both: findings for the known patterns, logs for the investigation.

DNSSEC validation

Protects what your workloads believe: for zones that are DNSSEC-signed, the resolver checks the chain of trust and discards responses that fail validation. Distinguish it from DNSSEC signing, which you enable on your own public hosted zone so that other resolvers can validate your records — one protects your clients, the other protects your users. Neither is a substitute for DNS Firewall: validation proves an answer is authentic, not that the domain is safe.

AWS IAM Identity Center

The modern answer to "manage human access across many AWS accounts" — and the reflex SCS-C03 rewards over creating IAM users per account. You connect an identity source (the Identity Center directory, AWS Managed Microsoft AD or AD Connector, or an external SAML 2.0 IdP with SCIM provisioning), author permission sets, and assign a permission set to a user or group in a specific account. Each assignment materialises as a protected IAM role named AWSReservedSSO_<permission-set>_<suffix> in that account, so every session is temporary and every grant is visible to IAM tooling. Deleting the last assignment deletes the role; a later re-assignment creates a role with a new suffix, which is why resource policies should reference the role path rather than a pinned ARN.

Permission set

Authored once in Identity Center, assigned many times: an account assignment pairs a permission set with a user or group and a target account, and Identity Center provisions the corresponding AWSReservedSSO_ role there. Editing a permission set does not push itself out — the assignment must be re-provisioned, which is the usual cause of "the policy looks right but access still fails". Permission sets are the natural RBAC unit for workforce access (one per job function), and they can carry a session duration and a permissions boundary.

Identity source (IAM Identity Center)

Three options, and the exam picks between them on where identity is mastered. The Identity Center directory is self-contained and fine for small organizations with no existing IdP. Active Directory (AWS Managed Microsoft AD, or AD Connector pointing at on-premises AD) suits a Windows estate that already owns the user lifecycle. An external identity provider (Entra ID, Okta, Ping) connects over SAML 2.0 for authentication plus SCIM for provisioning, and is the answer whenever the scenario says identities already live in a corporate IdP. Changing the identity source resets assignments, so this is an early design decision, not a late one.

SCIM provisioning

SAML carries authentication at sign-in time; SCIM (System for Cross-domain Identity Management) carries the directory itself. With SCIM enabled, creating, updating, or disabling a user in the IdP propagates into Identity Center, so group membership — and therefore every account assignment derived from it — tracks the HR system. Without it you federate successfully but manage group membership by hand, and deactivated employees keep their assignments. "Access must be revoked when the employee is offboarded in the IdP" is a SCIM requirement.

SAML 2.0 federation

For direct-to-IAM federation you create an IAM SAML identity provider from the IdP metadata document, write a role trust policy that trusts that provider (typically with StringEquals on SAML:aud = https://signin.aws.amazon.com/saml), and the IdP sends an assertion whose Role attribute carries role-ARN,provider-ARN pairs. The client calls AssumeRoleWithSAML — no AWS credentials needed — and gets a session of up to 12 hours, bounded by the assertion SessionDuration and the role ceiling. The assertion can also set RoleSessionName, PrincipalTag:<key> session tags, and TransitiveTagKeys. IAM Identity Center wraps all of this; hand-built SAML federation is what you do when you cannot use it.

OIDC / web identity federation

You register the provider as an IAM OIDC identity provider (its issuer URL and audience), then trust it in a role trust policy and constrain the token claims with conditions — pinning the aud claim and, for example, the sub claim so only one repository, branch, or Kubernetes service account may assume the role. AssumeRoleWithWebIdentity needs no AWS credentials, which is exactly the point: CI pipelines, mobile apps, and EKS pods authenticate with a signed token instead of a stored access key. Leaving sub unconstrained is the classic over-trust bug.

Cognito user pool

Sign-up and sign-in for your own application: hosted sign-in pages, password policies, MFA, token issuance (an OIDC ID token describing the user, an OAuth 2.0 access token for scoped API authorization, and a refresh token), threat-protection features such as compromised-credential detection and adaptive authentication, federation to social and enterprise IdPs, and Lambda triggers at every step of the flow (pre-sign-up, pre/post-authentication, pre-token-generation, custom auth challenges, user migration). A user pool token authorizes calls to your API — through an API Gateway Cognito authorizer, say. It does not call AWS APIs.

Cognito identity pool (federated identities)

The other half of Cognito, and the distinction the exam tests hardest. An identity pool accepts a token from a user pool, a social or enterprise IdP, a SAML or OIDC provider, or a developer-authenticated identity, and returns short-lived AWS credentials for an IAM role. It carries two role slots — an authenticated role and an optional unauthenticated (guest) role — and can pick between several roles by token claim or rules-based mapping. "Let signed-in mobile users upload directly to their own S3 prefix" is an identity-pool requirement; "let users sign up and sign in" is a user-pool requirement.

Cognito role mapping

Role-based access control inside the credential exchange. Token-based mapping reads a role claim (cognito:preferred_role / cognito:roles, populated from user-pool groups); rules-based mapping matches a claim such as custom:department against conditions and maps each match to a role; otherwise the pool issues the default authenticated role. The attribute-based alternative is "attributes for access control", which projects claims into principal tags so one role plus aws:PrincipalTag conditions replaces a role per group.

AWS Directory Service

AWS Managed Microsoft AD runs actual Microsoft Active Directory domain controllers managed by AWS: it supports schema extensions, Group Policy, and one-way or two-way forest trusts with your on-premises forest — the choice when AWS workloads need to be domain-joined or when you want AD without replicating users. AD Connector is a directory gateway that proxies authentication requests to an existing on-premises directory; it stores no directory data in AWS, supports no trusts, and is the lightest way to let on-premises users sign in to AWS. Simple AD is a small Samba 4 compatible directory for basic needs — no trusts, no MFA, no schema extensions. Pick by where identity is mastered and whether a trust is required.

aws:MultiFactorAuthPresent

Available only when the caller uses temporary credentials. It is present and true for a session created with MFA, present and false when AWS minted temporary credentials for an IAM user without MFA (console sign-in, or a service re-using the user’s credentials), and not present at all for requests signed with long-term IAM user access keys or for federated identities. That absence is the trap: "Bool": {"aws:MultiFactorAuthPresent": "false"} in a Deny never matches an access-key request, because a missing key cannot be evaluated. Use BoolIfExists to fail closed. For federated or Identity Center identities, carry the IdP’s authentication method in as a session tag and condition on that instead.

FIDO2 security key / passkey

AWS supports three MFA families: virtual TOTP authenticators, hardware TOTP tokens, and FIDO2 — security keys and platform passkeys. FIDO2 is the strongest because the credential is cryptographically bound to the AWS sign-in origin, so a proxy phishing page cannot replay it; a TOTP code can be read aloud to an attacker. Root users and IAM users can register multiple MFA devices, which is what makes hardware MFA practical without a single point of failure.

AWS account root user

The root user has unrestricted access to its account and is not constrained by identity policies, permissions boundaries, or (for the management account) SCPs — which is why the standing advice is to give it hardware MFA, no access keys, a locked-away password, and no day-to-day use. A short list of tasks genuinely requires it: closing the account, changing the account name, email address, or root password, changing the support plan, enabling S3 MFA Delete on a bucket, restoring access after an IAM policy locked everyone out, and deleting a resource policy that denies all principals. Everything else belongs to a role.

Centralized root access

Two independent capabilities, enabled in IAM for an organization. Root credentials management lets you delete (and later restore) the root user password, access keys, and MFA for member accounts, so there is no root credential to phish in hundreds of accounts. Privileged root actions lets the management account or an IAM delegated administrator call sts:AssumeRoot against a member account with a task policy ARN — for example arn:aws:iam::aws:policy/root-task/S3UnlockBucketPolicy — to perform one scoped task such as deleting a bucket or queue policy that locked everybody out. Sessions are short (15 minutes), root credentials themselves cannot make the call, the global STS endpoint is not supported, and every session is recorded in CloudTrail.

Break-glass access

Designed before it is needed: credentials or a role that nobody uses day to day, protected by MFA held under split custody, excluded from the automation that would otherwise rotate or delete it, and wired so that every use raises an alert — an EventBridge rule on root sign-in or on AssumeRoot, paging a human. The procedure names who may invoke it, what approval is required, and what must be written down afterwards. The test the exam applies: if your IdP is unavailable or an SCP has locked the account, can you still get in, and would you know within minutes that someone did?

AWS STS (Security Token Service)

Every temporary credential in AWS comes from STS as a triple — access key id, secret access key, and session token — with an expiry. AssumeRole is for an AWS principal taking a role (same account or cross-account). AssumeRoleWithSAML and AssumeRoleWithWebIdentity take an IdP assertion or OIDC token and need no AWS credentials at all. GetSessionToken re-issues an IAM user’s own permissions as a temporary, optionally MFA-validated session. GetFederationToken mints credentials for a federated user from an IAM user’s permissions, narrowed by a required session policy. AssumeRoot is the newer privileged-task session for centralized root access. Prefer the Regional endpoints and keep the global endpoint for legacy callers only.

AssumeRole

Two independent permissions must line up: the role’s trust policy must name the caller as a principal for sts:AssumeRole, and the caller’s identity policy must allow sts:AssumeRole on that role ARN. Both sides, always — there is no implicit trust in either direction. The resulting session carries the role’s permissions (narrowed by any session policy), a DurationSeconds between 15 minutes and the role’s maximum session duration, and a role session name that shows up in CloudTrail as the assumed-role ARN. Optional ExternalId, SourceIdentity, session tags, and policies are all set on this call.

Role trust policy

A trust policy is the one policy every role must have. It names principals — an account root (delegating the decision to that account’s administrators), a specific user or role ARN, an AWS service principal such as ec2.amazonaws.com, a SAML provider, or an OIDC provider — plus the sts actions they may call (sts:AssumeRole, sts:TagSession, sts:SetSourceIdentity) and conditions (sts:ExternalId, aws:PrincipalOrgID, aws:SourceArn, OIDC claim pins). It grants nothing inside the account: what the session can do comes from the role’s separate permissions policy. Confusing the two is the most common wrong answer in this domain.

Maximum session duration

Set on the role, not on the policy. AssumeRole accepts DurationSeconds from 900 seconds up to this ceiling, and fails if it asks for more. GetSessionToken and GetFederationToken have their own ranges (up to 36 hours for an IAM user caller; one hour for root). Shorter sessions shrink the window in which a stolen credential is useful but push re-authentication onto the workload, so the SDK credential providers refresh automatically — the exam’s preferred answer to "credentials keep expiring mid-job" is almost never "raise the ceiling to 12 hours".

Role chaining

Chaining is normal in multi-account designs (a pipeline role assumes a deployment role in the target account) but it changes the rules: the resulting session may not exceed one hour, and asking for a longer DurationSeconds makes the call fail outright. Session tags marked transitive survive the hop; non-transitive tags do not. sourceIdentity, once set, persists through the chain and cannot be changed. When a workload "needs an 8-hour session but only gets one hour", the diagnosis is almost always that it is chaining.

sourceIdentity

Passed as SourceIdentity on the assume-role call and visible as sts:SourceIdentity / aws:SourceIdentity in conditions and in CloudTrail. It requires sts:SetSourceIdentity permission in both the caller’s identity policy and the role’s trust policy, and once set it cannot be changed or removed for the life of the session — including through further role chaining. That immutability is the point: role session names can be chosen freely by the caller, so they are not evidence; sourceIdentity is. Contrast with session tags, which carry attributes for authorization rather than attribution for audit.

sts:ExternalId

When you grant a third-party vendor a role, their account becomes a deputy that could be confused into acting for someone else’s benefit: another of their customers names your role ARN and the vendor, holding valid credentials, assumes it. The fix is a StringEquals condition on sts:ExternalId in your trust policy, with a value the vendor associates with your account only. The vendor generates it (so one customer cannot guess another’s), you never put it in a cross-customer template, and the vendor must pass ExternalId on every assume-role call. It is not a password — it is an unguessable correlation id that pins the role to one tenant.

Confused deputy

The deputy is any party you have trusted to act for you: a SaaS vendor’s account, or an AWS service that assumes a role in your account to do work. The attack supplies the deputy with your resource identifiers so it applies its legitimate access to the wrong tenant. Three defences, each matched to a case: sts:ExternalId for a third-party account assuming your role; aws:SourceAccount and aws:SourceArn in a service-principal trust or resource policy, so the service may only act when the calling resource belongs to you; and aws:PrincipalOrgID to scope access to your own organization instead of listing account ids.

aws:SourceAccount / aws:SourceArn

When a service principal (s3.amazonaws.com, cloudtrail.amazonaws.com, sns.amazonaws.com, rolesanywhere.amazonaws.com) calls your role or resource, the request context carries which account and which resource triggered it. aws:SourceAccount pins the owning account; aws:SourceArn pins the exact bucket, trail, topic, or trust anchor — and supports ArnLike for a family of resources. Use ArnLike with a wildcard only when the resource name is not yet known at policy-authoring time. Pairing both keys is the documented pattern; aws:SourceArn alone already implies the account.

aws:PrincipalOrgID

The data-perimeter workhorse. A bucket policy, KMS key policy, or role trust policy conditioned on aws:PrincipalOrgID equal to o-exampleorgid admits every principal in that organization and nothing else, and it keeps working when accounts are added or closed. aws:PrincipalOrgPaths goes finer, matching an organizational-unit path so only principals under a given OU qualify. Both are principal keys, so they describe who is calling, not what is being called — the resource-side mirror is aws:ResourceOrgID.

IAM Roles Anywhere

You register a trust anchor (a certificate authority — AWS Private CA or your own CA’s certificate), create a profile listing the roles those certificates may assume plus an optional session policy, and write a role trust policy that trusts the rolesanywhere.amazonaws.com service principal with aws:SourceArn pinned to the trust anchor. The workload’s signing helper calls CreateSession with its certificate and receives standard temporary credentials. Certificate attributes arrive as principal tags (aws:PrincipalTag/x509Subject/CN, x509Issuer/CN, x509SAN/...), so you can condition on them, and the certificate serial number becomes the role session name in CloudTrail. Revocation is the CA’s job — publish a CRL to the trust anchor.

Trust anchor (Roles Anywhere)

Either an AWS Private CA in your account or an external CA whose certificate you upload. Every end-entity certificate presented to CreateSession must chain to a trust anchor, be X.509v3 with CA: false, carry digital-signature key usage, and be signed with SHA-256 or stronger. The trust anchor ARN is what role trust policies pin with aws:SourceArn, so one compromised CA cannot be swapped for another, and a certificate revocation list attached to the trust anchor is how you cut off a decommissioned host.

S3 presigned URL

The signer computes a SigV4 signature over the request (method, bucket, key, expiry) using their own credentials; S3 validates the signature and then authorizes the request as that principal. Two consequences the exam tests. First, a presigned URL cannot exceed the signer’s own permissions — presigning a GetObject you are not allowed to perform produces a URL that fails. Second, it is a bearer token: no further authentication happens, so anyone who obtains the link has that access until expiry. Lifetimes: up to 7 days when signed with IAM user credentials, 1 minute to 12 hours from the console, and never longer than the temporary credentials used to sign it — deleting or revoking those credentials invalidates the URL immediately. Presigned POST does the same for browser uploads with policy-bound form fields.

Session policy

Supplied on the STS call as one inline JSON document and/or up to ten managed policy ARNs (the plaintext is limited to about 2,048 characters). The resulting session’s permissions are the intersection of the role’s identity-based policy and the session policy — so a session policy can only narrow, never add. This is how a multi-tenant broker hands one role out safely: the same role, scoped per request to one customer’s prefix. GetFederationToken requires one. An IAM administrator cannot see it after the fact, which is why session scoping belongs in code review as well as in policy review.

Session tag

Passed on AssumeRole as Tags, or carried in a SAML assertion (PrincipalTag:<key>) or an OIDC token claim, and requiring sts:TagSession in the role’s trust policy. Once set, aws:PrincipalTag/project can be compared against aws:ResourceTag/project so one policy serves every project without being rewritten. Session tags override identical role tags for that session. They carry attributes for authorization; sourceIdentity carries attribution for audit — do not use one for the other’s job.

Transitive session tag

List the keys in TransitiveTagKeys on the assume-role call and those tags persist into every subsequent session in the chain, where they cannot be overwritten. Without it, ABAC conditions that worked on the first hop silently fail on the second — one of the quieter causes of "the policy is right but access is denied" in multi-account pipelines. Keep the transitive set small: a transitive tag is permanently part of the session’s identity for the rest of the chain.

Policy evaluation logic

AWS starts from an implicit deny, gathers every applicable policy, and checks for an explicit Deny — in an organization policy (SCP or RCP), a resource-based policy, a permissions boundary, a session policy, or an identity-based policy. Any one of them ends the request. With no explicit deny, the request is allowed only if it survives each layer that applies: organization policies must permit it, the boundary must permit it, the session policy must permit it, and an Allow must exist in the identity-based policy — or, within the same account for resource-policy services, in the resource-based policy instead. Cross-account requests are evaluated twice, once in each account, and both evaluations must return Allow.

Implicit deny

Every request starts denied. An implicit deny is simply the absence of a matching Allow, which is why adding a policy can fix an implicit deny but never an explicit one. The distinction matters when reading a failure: "no identity-based policy allows kms:Decrypt" is an implicit deny you fix by granting; "an SCP denies kms:* outside approved Regions" is an explicit deny you fix by changing the guardrail or the request. CloudTrail’s errorMessage and the decoded authorization failure message tell you which you are looking at.

Resource-based policy

S3 bucket policies, KMS key policies, SQS queue policies, SNS topic policies, Lambda function policies, Secrets Manager secret policies, ECR repository policies, and IAM role trust policies are all resource-based. Two properties follow from the Principal element. Within the same account, either the identity policy or the resource policy can supply the Allow. Across accounts, the resource policy identifies who may access the resource, and the caller still needs an Allow in its own identity policy — both accounts must agree. AWS KMS is the service that breaks the pattern: its key policy is mandatory, and IAM policies alone cannot authorise key use.

Permissions boundary

The effective permissions of a principal with a boundary are the intersection of its identity-based policies and the boundary. That makes it the safe way to delegate IAM: give a developer iam:CreateRole and iam:AttachRolePolicy, but condition those actions on iam:PermissionsBoundary so every role they create must carry your boundary — they can build roles, but none more powerful than the boundary allows. Boundaries do not limit resource-based policy grants to the principal, and they do not apply to service-linked roles. A request denied only by the boundary is the classic "my identity policy clearly allows this" puzzle.

Service control policy (SCP)

Attached to the organization root, an OU, or an account, and inherited downward: a request must be permitted by every SCP in the path and by an IAM policy. SCPs affect only IAM users and roles in member accounts — the management account is unaffected, as are service-linked roles. The default FullAWSAccess SCP is an allow-list ceiling; replacing it with a narrower allow-list, or layering Deny statements with condition keys, are the two authoring styles. Because an SCP grants nothing, "the SCP allows it" is never a complete answer to why a request succeeded.

Resource control policy (RCP)

Where an SCP constrains your principals, an RCP constrains your resources: attach one to an OU and every bucket, key, queue, or secret underneath rejects access that the RCP denies, even from a principal outside your organization. The canonical use is a data perimeter — deny all access unless aws:PrincipalOrgID matches, with carve-outs for named AWS service principals. RCPs grant nothing. One structural difference from SCPs is worth carrying into the exam: an SCP applies across all AWS services, while an RCP applies only to the services that support it — a large and steadily growing subset, so confirm the current list in the AWS Organizations documentation rather than assuming a particular service is covered. The exclusions are the durable part: RCPs never restrict any action on resources in the management account, never affect the effective permissions or trust policy of a service-linked role, do not apply to AWS managed KMS keys, and do not restrict kms:RetireGrant.

NotAction

"Effect": "Allow", "NotAction": "s3:DeleteObject", "Resource": "*" does not mean "all S3 except delete" — it means every action in every AWS service except that one, including services launched after the policy was written. Used with Deny it is the honest tool: deny everything except the handful of actions a break-glass or MFA-setup flow needs. NotResource behaves the same way, and NotPrincipal is worse still — with Deny it produces policies that are very hard to reason about, and AWS recommends a Deny with a condition on aws:PrincipalArn instead. On the exam, NotAction plus Allow in a stem is usually the over-permission you are being asked to spot.

Policy variable

Variables turn one statement into per-principal scoping: "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/home/${aws:username}/*" gives every IAM user their own prefix with no per-user policy. They work in Resource and in condition values, and a variable whose key is absent from the request context does not match — so a policy relying on ${aws:PrincipalTag/team} fails closed for a principal with no team tag. Default values (${aws:PrincipalTag/team, "none"}) exist for that case. ${aws:username} is meaningless for role sessions, which is why ABAC on tags scales where username variables do not.

Condition key

Global keys (prefix aws:) apply across services: principal keys (aws:PrincipalArn, aws:PrincipalOrgID, aws:PrincipalTag/...), resource keys (aws:ResourceTag/..., aws:ResourceOrgID), request keys (aws:RequestTag/..., aws:TagKeys, aws:RequestedRegion), network keys (aws:SourceIp, aws:SourceVpc, aws:SourceVpce, aws:VpcSourceIp, aws:SecureTransport), and call-path keys (aws:ViaAWSService, aws:CalledVia). Service keys (s3:..., kms:..., ec2:...) apply only to that service’s actions. The rule that catches people: a key absent from the request context causes the condition not to match, so a Deny you expected to fire may simply not apply — which is what the ...IfExists operators exist to handle.

…IfExists condition operator

Without the suffix, a condition on a key that is absent from the request context does not match, so the statement does not apply. With it, the condition is satisfied when the key is missing. That is how MFA enforcement is written to fail closed: "BoolIfExists": {"aws:MultiFactorAuthPresent": "false"} in a Deny catches both the non-MFA session (key present and false) and the long-term access-key request (key absent). The same suffix is how you write "requests through this VPC endpoint, or requests where the key does not apply" without accidentally exempting whole classes of caller.

ForAllValues / ForAnyValue

Multi-valued keys such as aws:TagKeys or aws:CalledVia carry a list. ForAllValues:StringEquals on aws:TagKeys is the tag allow-list pattern — every key in the request must be one you named, which is how you stop a user from attaching tags you do not control. ForAnyValue is the "at least one of these" test. The subtlety: ForAllValues evaluates to true when the key is absent from the request entirely (an empty set vacuously satisfies "all"), so pairing it with a Deny and no Null check can leave untagged requests permitted.

ABAC (attribute-based access control)

One statement, conditioned on aws:PrincipalTag/project equalling aws:ResourceTag/project, grants every team access to exactly its own resources; tag a new resource correctly and it is governed on creation. The pattern needs three things to be safe: tags that cannot be set by the principals they govern (control tag:TagResource, and condition aws:RequestTag/... and aws:TagKeys on create), attributes delivered reliably into the session (session tags from SAML, OIDC, or Identity Center attributes for access control), and a deny for untagged resources so "no tag" does not mean "no restriction". ABAC scales with resource count; RBAC scales with job-function count.

RBAC (role-based access control)

The familiar model, and still the right one where the job functions are few and the permissions are genuinely different per function — an auditor, a database administrator, a release engineer. Implemented with IAM Identity Center permission sets assigned to groups, or IAM roles with per-function policies. Its cost is maintenance: every new project, bucket, or account means editing policies, and policies drift toward wildcards to avoid the churn. Most real estates are hybrid — RBAC for the coarse function, ABAC conditions inside it for the per-project scope.

aws:RequestTag / aws:TagKeys

aws:RequestTag/<key> is the value passed for that key in a create or tag call; aws:TagKeys is the multi-valued list of keys being set. Together they make tag-on-create enforceable: require aws:RequestTag/project to equal the caller’s aws:PrincipalTag/project, and use ForAllValues:StringEquals on aws:TagKeys to allow only an approved key set. Without this, ABAC is self-service — a principal simply tags a resource with another team’s project value and inherits their access. Contrast aws:ResourceTag/<key>, which describes tags already on the target resource.

IAM Access Analyzer

Three analyzer types, each answering a different question. External access examines resource-based policies — including role trust policies — against a zone of trust (an account or the whole organization) and reports anything reachable from outside it, whether or not anyone has actually used it; intended sharing gets archived with an archive rule. Unused access continuously tracks roles, users, credentials, and permissions and reports what has gone unused beyond your chosen tracking period, with policy recommendations that narrow the grant. Internal access shows which principals inside your organization can reach a nominated critical resource. Alongside them: policy validation findings while you author, policy generation from CloudTrail history, and custom policy checks for CI.

External access findings

Generated by analysing resource-based policies, not access logs: a finding says a bucket policy, key policy, queue policy, or role trust policy permits an outside principal, regardless of whether anyone used it. The zone of trust is the account or organization you chose when creating the analyzer, so intra-organization sharing produces no noise once the scope is the organization. Analysis re-runs when a policy changes (allow up to about 30 minutes). Deliberate sharing — a partner’s audit role — is handled with archive rules so the live finding list stays meaningful.

Unused access findings

Create an unused access analyzer with a tracking window (match it to your audit cycle; six months or a year is a sensible start) and it reports unused roles, unused IAM user access keys and passwords, and unused permissions — services and actions granted but never called — across the account or organization, with recommended narrower policies. Two cautions the exam respects: destructive or seasonal permissions may legitimately go unused for months, and tagged exception principals can be excluded from evaluation rather than forced into a false positive.

Access Analyzer policy generation

Point it at a role or user and a CloudTrail trail covering a representative window, and it produces a policy listing the services and actions that principal really used, with resource-level detail where the service supports it. It is a starting draft, not an answer: anything the principal did not do during the window is missing, so the workflow is generate, review against intent, add what is seasonal, then deploy and watch. The exam’s preference throughout this domain is evidence over guessing — generated policies and last-accessed data over hand-written wildcards.

Custom policy check

Policy review as a build step. CheckNoNewAccess compares a proposed policy against the existing one and fails if the change grants anything new — the guardrail for "this edit was meant to narrow, not widen". CheckAccessNotGranted fails if a named set of actions or resources is reachable at all, which is how you encode "nothing may grant iam:PassRole on the admin role" or "no policy may allow kms:ScheduleKeyDeletion". CheckNoPublicAccess fails if a resource policy would expose the resource publicly. They run from the CLI, the SDK, or an IDE, which is what makes them CI controls rather than review opinions.

IAM policy simulator

Run in Principal mode against an existing user or role, or Custom mode against pasted policies before they are attached anywhere. It evaluates identity-based policies, permissions boundaries, resource-based policies, and SCPs — including condition keys and resource scoping — and returns a binary allow or deny per action and resource. Known gaps, all worth remembering: it does not evaluate RCPs, it makes no real request and uses no production context values, and results can diverge for VPC endpoint policies, role chaining, and multiple resource policies on one resource. It is the "will this break?" tool before deploying, and the "why was this denied?" tool after.

Last accessed information (Access Advisor)

Shown per user, role, group, or policy in the IAM console and available through GetServiceLastAccessedDetails. It answers "is this grant doing any work?" directly from AWS’s own tracking rather than from your reading of CloudTrail, covers a trailing window, and for supported services goes down to action-level last-used data. Organization-wide reports can be generated from the management account to find unused service permissions across accounts. Pair it with Access Analyzer unused access findings: last-accessed data tells you what was used, generated policies tell you what a minimal grant would look like.

Amazon Verified Permissions

IAM authorises AWS API calls; Verified Permissions authorises things only your application knows about — may this user comment on that document, may this manager approve that expense. You create a policy store, optionally define a schema describing your principal, resource, and action types, write Cedar policies (static, or templates instantiated per resource), and call IsAuthorized (or IsAuthorizedWithToken with a Cognito user-pool token, or BatchIsAuthorized for many decisions at once) with principal, action, resource, and context. The decision comes back ALLOW or DENY with the determining policies, so authorization leaves the application code and becomes reviewable, versioned policy.

Cedar

Cedar policies read close to the sentence they encode — permit(principal in Group::"editors", action == Action::"update", resource in Folder::"drafts") when { context.mfa == true }; — and the language is designed for automated reasoning, so tools can analyse what a policy set permits rather than only test samples. forbid beats permit, mirroring IAM’s explicit-deny rule. Cedar is application-level: it knows your entity types and attributes, not AWS service actions.

IAM path

Set at creation and not changeable afterwards, a path turns ARNs into a hierarchy: arn:aws:iam::111122223333:role/teams/payments/deploy. Policies can then delegate over a whole branch — allow iam:PassRole on arn:aws:iam::111122223333:role/teams/payments/*, or let a team administrator manage only entities under their own path — without enumerating names or relying on naming conventions a typo can break. The pattern pairs naturally with permissions boundaries for delegated administration, and it is how IAM Identity Center keeps its generated roles out of the way, under /aws-reserved/sso.amazonaws.com/.

Envelope encryption

The pattern behind nearly every “encrypted with KMS” claim on AWS. The client calls GenerateDataKey, receives a data key in plaintext and encrypted form, encrypts the payload locally with the plaintext copy, discards that copy, and stores the encrypted data key next to the ciphertext. Decryption sends only the small encrypted data key back to KMS. Three consequences the exam tests: your plaintext never reaches KMS, large objects never cross the KMS API (the Encrypt operation accepts at most 4 KB of plaintext), and decryption costs one small call regardless of object size.

AWS KMS Developer Guide — concepts

Data key

KMS does not store data keys — it generates them and forgets them, which is why the encrypted copy must be kept with the ciphertext. GenerateDataKey returns both forms, for a component that will encrypt immediately; GenerateDataKeyWithoutPlaintext returns only the encrypted form, for a component that must prepare envelopes it is not allowed to open. The plaintext data key is the one secret your own code is responsible for: hold it for the operation, overwrite the buffer, never log it or persist it. Lose the encrypted copy and the ciphertext is unrecoverable.

Encryption context

Supplied at encrypt time and required again, identical, at decrypt time — so a ciphertext created with {"tenant":"acme"} cannot be decrypted by a caller claiming another tenant, even when both may use the key. The pairs appear in CloudTrail, turning opaque Decrypt events into a searchable audit trail, and they can be matched in policy with the kms:EncryptionContext: condition keys, which lets one key serve many tenants with per-tenant authorization. Never put secrets in it: the values are logged in plaintext.

KMS key policy

Unlike most AWS resource policies, a key policy gates IAM rather than merely adding to it. The statement that makes IAM effective names the account root as principal with kms:* — meaning “principals in this account may be authorized by their IAM policies”. Remove it and the key is governable only by statements written directly in the key policy, which locks out even administrators: the classic troubleshooting scenario where AdministratorAccess gets AccessDenied with nothing misconfigured. The mirror problem is a key policy that delegates to the account root, so reviewing the key policy alone tells you nothing about who can actually decrypt.

AWS KMS Developer Guide — key policies

KMS grant

A grant names a grantee principal, the exact operations allowed, and optionally an encryption context constraint; it takes effect within seconds and ends with RetireGrant or RevokeGrant. AWS services create grants on your behalf constantly, which is why an encrypted EBS volume or an Aurora cluster keeps working after the creating session ends — and why the condition key kms:GrantIsForAWSResource exists, allowing kms:CreateGrant only for grants created by integrated AWS services. The exam tell for a grant is “temporary”, “programmatic”, “for the duration of the job”, or “without modifying the key policy”.

kms:ViaService

Written as "kms:ViaService": ["s3.eu-west-1.amazonaws.com"], it means the key may be used when S3 calls KMS on the principal’s behalf, but not when the principal calls Decrypt directly. That is credential-leak containment: stolen role credentials cannot be used to bulk-decrypt outside the service path the workload actually uses. The service names are Regional endpoints, so a multi-Region workload lists one per Region.

AWS managed key

Created the first time you use a service’s default encryption, one per service per Region. You see it and you see its use in your CloudTrail, but you cannot modify the key policy, cannot choose the rotation period (AWS rotates the material every year), and cannot share it with another account — which is why a snapshot encrypted with an AWS managed key cannot be shared at all, and must first be copied onto a customer managed key. There is no monthly key charge; you pay per request. Any requirement naming policy control, cross-account use, or a specific rotation period forces a customer managed key.

AWS owned key

The quietest of the three key types: it protects resources in services that encrypt by default without surfacing a key to you. You cannot view it, audit its use in your own trail, apply a policy to it, or rotate it — rotation belongs to the owning service. For exam purposes it is the answer only when the requirement is “encrypted at no cost with no key management”, and the distractor whenever auditability or control is mentioned.

KMS key rotation

Automatic rotation is optional for customer managed symmetric keys with AWS-generated material — annual by default, with a configurable period — and AWS managed keys rotate every year regardless. The key id, ARN, aliases and policies never change, and no application or ciphertext is affected, which is why rotation is not a revocation mechanism: old data does not become unreadable. Automatic rotation is unavailable for asymmetric keys, HMAC keys, keys in a custom key store, and keys with imported material.

AWS KMS Developer Guide — rotating keys

On-demand key rotation

Supported for symmetric encryption keys with AWS-generated material and, since 2024, for keys with imported material (EXTERNAL origin) — for which it is the only rotation path, because automatic rotation needs material KMS can generate. For imported material you first import the replacement, which waits in a pending-rotation state, then invoke the rotation. There is a cap of 25 on-demand rotations per key, it does not disturb an existing automatic schedule, and it is not available for asymmetric keys, HMAC keys, or keys in a custom key store. For multi-Region keys, import the same new material into the primary and each replica, then rotate on the primary.

Scheduled key deletion

KMS refuses to delete a key immediately because the consequence is absolute: when the window expires, the material is gone and every ciphertext under it is permanently unrecoverable, with no AWS support path. The default window is 30 days and the minimum is 7. The reversible alternative is DisableKey, which stops all cryptographic use at once and is undone by EnableKey — so read requirements carefully: “immediately prevent use, restorable later” is disable, “remove permanently after a mandatory waiting period” is scheduled deletion. Check CloudTrail and the key’s usage metrics before scheduling anything in production, because the failure surfaces months later at restore time.

Multi-Region key (MRK)

Created with ReplicateKey from a multi-Region primary. The shared material and key id make related keys interoperable, which is what allows cross-Region replication, cross-Region backup copies and DR sites to read the same ciphertext with no call into the origin Region. What stays separate matters equally: each replica is a real key in its own Region with its own key policy, grants, tags, aliases and enablement state, so authorization is per Region. Rotation is coordinated from the primary. Because one key material now spans Regions, AWS advises single-Region keys unless a concrete cross-Region requirement exists.

AWS KMS Developer Guide — multi-Region keys

AWS CloudHSM

Chosen when a regulation or contract demands sole custody of key material, or when an application must speak PKCS#11, JCE or the OpenSSL dynamic engine directly — TLS offload, custom PKI, bespoke cryptography. The trade is operational: you size the cluster across Availability Zones, manage crypto officer and crypto user credentials, take and keep backups, and own availability. Note the exam trap: “FIPS 140-3 Level 3” on its own does not justify CloudHSM, because AWS KMS also uses validated HSMs. The differentiators are single tenancy, sole custody and direct HSM protocol access.

KMS custom key store

The bridge between CloudHSM custody and KMS integration: S3 SSE-KMS, EBS, RDS, Secrets Manager and every key policy continue to work, but the material lives in and the operations happen in your single-tenant cluster. The give-ups are specific and testable — keys in a custom key store support neither automatic nor on-demand rotation (rotation is manual), and your cluster’s health becomes the key’s availability. Choose it when a question combines “we must control the HSMs” with “and the AWS service integrations must keep working”.

External key store (XKS)

The answer to a mandate that key material must never reside in AWS — typically an on-premises HSM or key manager. KMS keys appear and behave normally to callers, but every encrypt and decrypt depends on your proxy, so its availability, latency and monitoring become production infrastructure: an unreachable proxy means unreadable data. Treat the proxy as a tier-0 dependency with the same rigour as a database, and recognise XKS in a question only when the requirement is literally about key material leaving AWS — not merely about control or FIPS validation.

Imported key material (BYOK)

The ceremony: create the key (state PendingImport, unusable), call GetParametersForImport for a public wrapping key and a single-use import token that expire in 24 hours, generate and wrap your material, then ImportKeyMaterial with an optional ValidTo expiry. The differences from AWS-generated material are what the exam tests: no automatic rotation (on-demand only, after importing replacement material), AWS cannot recover it so your off-AWS copy is the only backup in existence, the material can be deleted with DeleteImportedKeyMaterial immediately and with no waiting period or expire on its own, and re-importing the identical material restores access while different material leaves old ciphertext unreadable forever. Everything else — policies, grants, CloudTrail, service integrations — behaves normally.

AWS KMS Developer Guide — importing key material

AWS Certificate Manager (ACM)

Prove domain control by DNS validation — a CNAME you publish once, which lets ACM re-validate and renew silently forever — or by email validation, which is fragile and makes renewals a human task. The private key for a standard public certificate stays inside ACM. Two Regional rules decide many exam questions: a certificate for CloudFront must be in us-east-1, and every other integration needs the certificate in the same Region as the resource, with no cross-Region copying or moving. Certificates imported into ACM from a third-party CA do not auto-renew — ACM only notifies.

AWS Certificate Manager User Guide

ACM exportable public certificate

Introduced on 17 June 2025, and it revises the old reflex that ACM certificates can never leave AWS. Export must be chosen at request time — it cannot be added afterwards, and certificates issued before that date can never be exported. Exportable certificates are valid for 395 days and carry a charge at issuance and at each renewal; the export yields the certificate, the chain and the private key encrypted under a passphrase you supply. The default public certificate remains free, auto-renewing and non-exportable, which is still the right answer whenever AWS terminates TLS for you.

AWS Certificate Manager FAQs — exportable public certificates

AWS Private Certificate Authority

Build a root CA that signs nothing but subordinates and subordinate CAs that do the day-to-day issuing, so an issuing CA can be revoked without replacing the whole trust chain. ACM can issue and renew private certificates from your CA for the same AWS services, and private certificates can be exported because your own CA signed them. Revocation must be designed at CA creation — CRL distribution to an S3 bucket, or an OCSP responder — because clients that cached a certificate without a distribution point will never look for one. Cost shape matters: a monthly charge per CA plus a charge per issued certificate, which is why “one CA per team” gets expensive and shared hierarchies are the norm.

Mutual TLS (mTLS)

The usual implementation on AWS is private certificates from AWS Private CA, issued per service or per device. Two designs are worth distinguishing: an Application Load Balancer can perform mTLS itself, validating client certificates against a trust store — the better answer when the requirement is “authenticate clients with certificates” — while a Network Load Balancer in TCP pass-through mode leaves the handshake to the application, which is required when the application itself must inspect the client certificate. API Gateway supports mTLS on custom domain names, and IoT Core uses per-device certificates as its native identity.

aws:SecureTransport

Offering HTTPS is not requiring it: S3, SQS, SNS, Secrets Manager and KMS all accept plaintext requests until a policy refuses them. A statement with "Effect": "Deny", "Principal": "*" and "Condition": {"Bool": {"aws:SecureTransport": "false"}} closes the door regardless of what identity policies allow, because an explicit deny beats every allow. For S3, include both the bucket ARN and the /* object ARN or half the API stays uncovered. At organization scale the same statement belongs in a resource control policy so no account can create a resource without it.

s3:TlsVersion

Written with a numeric operator — "NumericLessThan": {"s3:TlsVersion": 1.2} in a deny statement — it enforces a protocol floor on the bucket itself rather than on a load balancer. Deploy it carefully: old SDKs and embedded clients fail immediately and, from their point of view, silently. CloudTrail records the TLS version of each request, which makes a dry-run audit possible before enforcement.

ELB security policy

Selected per HTTPS or TLS listener. Choose a policy whose minimum protocol version matches your standard (the TLS 1.2-and-above and TLS 1.3 families) and the load balancer refuses anything weaker during negotiation, before any request exists to inspect. It is a listener property, not a policy document, which is the distinction a question tests when it offers aws:SecureTransport as an alternative. Removing the HTTP listener — or leaving one whose only action is a redirect to HTTPS — is the complementary step.

TLS termination

Three patterns with three different answers. Terminate at CloudFront or an ALB: one certificate to manage and full layer-7 inspection (WAF, headers, routing), but load-balancer-to-target traffic is plaintext. Re-encrypt to the target with an HTTPS target group: every hop encrypted, targets can use private certificates because the load balancer does not require a public chain — the answer whenever a requirement says end-to-end or names PCI DSS or HIPAA. Pass through on an NLB TCP listener: the load balancer holds no key and sees no payload, required when the application itself must terminate TLS, at the cost of all AWS layer-7 inspection.

HTTP Strict Transport Security (HSTS)

Added without touching application code through a CloudFront response headers policy, which sets Strict-Transport-Security with a max-age and optional includeSubdomains and preload. It protects the next request rather than the first, so it complements rather than replaces a redirect and an explicit deny. A long max-age with includeSubdomains is effectively irreversible for its duration, so confirm every subdomain can serve HTTPS before enabling it.

AWS PrivateLink

Two shapes: a gateway endpoint (S3 and DynamoDB only, implemented as a route table entry) and an interface endpoint (an ENI with a private IP address for a specific service). PrivateLink also lets you expose your own service behind a Network Load Balancer so consumers in other VPCs or accounts reach it through their own interface endpoints, with no peering and no shared routes. Two things it does not do: it does not encrypt anything — traffic to AWS APIs is TLS either way — and it does not disable the service’s public endpoint, which stays usable with the same credentials until a resource policy condition closes it.

Gateway endpoint

Because it works by routing rather than by a network interface, it serves only requests that originate inside the VPC: it cannot carry traffic arriving from on premises over Direct Connect or VPN, which is the row that decides most endpoint questions. It has no security group, so restricting who in the VPC may use it is done with route tables, the endpoint policy and IAM. Available for S3 and DynamoDB and no other service.

Interface endpoint

The general PrivateLink mechanism: deploy an ENI per subnet, enable private DNS so the service’s normal hostname resolves to it with no code change, and control connectivity with a security group like any other ENI. It works for hundreds of AWS services and for services published through PrivateLink, and — unlike a gateway endpoint — it serves callers arriving over Direct Connect or VPN. Charged hourly per Availability Zone plus per gigabyte processed.

VPC endpoint policy

Its default is full access, so tightening it is a deliberate act. An endpoint policy allowing only your own buckets, or only resources belonging to aws:PrincipalOrgID, means a compromised instance cannot push data to an attacker’s bucket through your endpoint. It can never grant anything: the resource policy and the identity policy must still allow the call. Distinguish it from the mirror control — a resource policy condition on aws:SourceVpce or aws:SourceVpc, which requires that requests arrive through the endpoint and is what stops the same credentials working from a laptop.

AWS Client VPN

Users authenticate with certificates, Active Directory or a SAML identity provider and receive a network position: an IP address and routes. That makes it the right answer when access is network-shaped — a database port, a non-HTTP protocol, a subnet an engineer must reach. The contrast the exam draws is with AWS Verified Access, which publishes individual applications and evaluates every request instead of granting a tunnel.

AWS Verified Access

Applications are published individually, and policies are written over identity (from IAM Identity Center or an OIDC provider) and device trust (from a trust provider such as an endpoint-management product), so a user on a non-compliant laptop is refused even with valid credentials. Because each request is evaluated rather than one authentication granting hours of network reach, it is the answer when a requirement mentions device trust, per-application access, or removing the VPN client from the user experience.

Nitro System encryption in transit

Authenticated encryption with 256-bit keys, applied by the Nitro card to instance-to-instance traffic and to traffic between an instance and its EBS volumes or supported file systems. The qualifiers are strict and are what the exam tests: supported instance types only, same Region, same VPC or a peered VPC, and no intermediate hop — a transit gateway, NAT gateway, load balancer or gateway endpoint in the path means the hardware encryption does not apply, silently. It is a strong baseline, not a substitute for application TLS, and “we run Nitro instances” is not a complete answer to an encryption-in-transit requirement.

Amazon EC2 User Guide — encryption in transit

AWS Nitro Enclaves

Built for processing that must be unreadable even to the parent instance’s own operators and root user: decrypting payment data, handling private keys, running a model over sensitive inputs. Cryptographic attestation lets KMS verify the enclave’s measurements before releasing a key, through a key policy condition on the attestation document, so a key can be made usable only inside an approved enclave image. Distinguish it from Nitro encryption in transit: enclaves isolate processing, Nitro encryption protects links. A question about isolating sensitive processing means enclaves.

EMR security configuration

In-transit encryption covers Spark shuffle, block transfer, the Hadoop RPC layer and the web UIs, and it requires certificates you supply: a zip of PEM files in Amazon S3, or a custom certificate provider class. At-rest encryption is a separate switch in the same configuration, so enabling one does not enable the other. The configuration is referenced when the cluster is created and cannot be attached to a running cluster — which is why “encrypt the existing cluster’s shuffle traffic” means launching a new cluster.

Inter-container traffic encryption (SageMaker AI)

Set at job creation and not changeable mid-job. It is off by default because it can measurably slow large distributed training, particularly deep learning at scale — an explicit cost-versus-compliance choice rather than an oversight, and the reason a question about a training job slowing down after a security change usually points here. The analogous controls elsewhere are the EMR security configuration and, for Kubernetes, a service mesh with mTLS.

EKS secrets envelope encryption

Configured as a KMS provider on the cluster and enablable on existing clusters, after which the KMS key becomes a hard dependency of the cluster. It is one of three distinct EKS concerns the exam keeps separate: control-plane TLS is always on and says nothing about workloads; pod-to-pod traffic is encrypted only by a service mesh with mTLS, an encrypting CNI, or Nitro hardware encryption where the instance types and path qualify; and Secrets at rest need this envelope encryption. “EKS uses TLS, so everything is encrypted” is the claim to reject.

VPC Encryption Controls

As of October 2026 this is the control for an auditor who asks not “can it be encrypted?” but “prove nothing unencrypted is possible”. Monitor mode enriches VPC Flow Logs with an encryption-status field and identifies resources that could allow unencrypted communication; enforce mode blocks the creation or attachment of non-compliant resources and restricts network interfaces to encryption-capable instance types. It can be applied per VPC, per account, or across an organization through declarative policies.

SSE-S3

Requires no configuration and adds no request cost. What you give up is everything that depends on owning the key: you cannot audit individual key use, cannot apply a key policy as a second gate, and cannot revoke access to the objects by disabling a key. It is the correct and cheapest answer when the requirement is simply “data must be encrypted at rest” — and the distractor whenever auditability, key control or revocation is mentioned.

SSE-KMS

Those three properties are the reasons to pay for it, and each maps to a requirement phrase the exam reuses — “audit trail of key usage”, “separation of duties between bucket owner and key owner”, “revoke access to all objects immediately”. The cost is a KMS request per object operation, which is what S3 Bucket Keys exist to reduce. Note the authorization consequence: a caller must be allowed by the bucket policy and authorized on the key, so cross-account reads need the key policy to name the external principal — the source of the most confusing AccessDenied in S3.

S3 Bucket Keys

One bucket setting, applies to new objects, requires no application change, and preserves full KMS key control and CloudTrail auditability. It is the designed answer to “our KMS bill exploded after enabling encryption” on a high-volume bucket, and there is rarely a reason not to enable it. One exception worth remembering because it makes a clean exam question: Bucket Keys are not supported for DSSE-KMS.

Amazon S3 User Guide — reducing the cost of SSE-KMS with S3 Bucket Keys

DSSE-KMS

It exists for a narrow set of regimes — certain national-security and defence requirements — that mandate two layers at rest. It costs more per request than SSE-KMS and cannot use S3 Bucket Keys, so at volume it is the most expensive option. Select it only when a requirement names dual-layer encryption explicitly; otherwise it is a distractor offered against SSE-KMS.

SSE-C

The cryptography is AWS’s, the key custody is entirely yours — including the consequence that losing the key makes the object unrecoverable with no support path. Every request must use HTTPS, for the obvious reason that the key travels in headers. S3 features and management operations that need to read object content cannot work with SSE-C objects. Choose it when the requirement is “AWS must not store our key”; choose client-side encryption when it is “AWS must never be able to decrypt our data”.

Client-side encryption

The only option that makes “AWS cannot decrypt our data under any circumstances” literally true, because no usable key and no plaintext ever reach the service. Keys can come from KMS, CloudHSM or your own key manager, and the message format carries the wrapped data key with the ciphertext. The price is that key management, key rotation, library upgrades and the consequences of a lost key all move to you — budget for that operational weight before choosing it over SSE-KMS.

S3 Block Public Access (BPA)

The four settings block new public ACLs, ignore all existing public ACLs, block new public bucket policies, and restrict public and cross-account access through any public policy. When on, a bucket policy granting Principal: "*" simply has no effect — which is what makes BPA an interlock rather than another policy to reason about. All four are enabled by default on new buckets since April 2023, so the modern failure mode is someone switching them off. When content genuinely must be public, serve it through CloudFront with origin access control or with presigned URLs and leave the interlock in place.

S3 Object Ownership (bucket owner enforced)

The default for new buckets and the modern answer to the old cross-account problem where objects uploaded by another account stayed owned by the uploader and were unreadable to the bucket owner. With bucket-owner-enforced, ACLs are not evaluated at all, which removes a whole class of accidental exposure and a whole class of confusing permission puzzles. If a question describes objects the bucket owner cannot read after a cross-account upload, it is describing the pre-enforced world and this is the fix.

S3 access point

Each access point has its own hostname and access-point policy, so fifteen teams get fifteen narrow policies while the bucket policy delegates to access points. An access point can never grant more than the underlying bucket policy permits, making it a delegation and simplification mechanism rather than a bypass. Multi-Region Access Points add a single global hostname that routes to the nearest or healthiest bucket in a set, with failover — global routing, not per-team policy separation, which is the distinction a question tests.

S3 Object Lock

A bucket-level flag — settable at creation or on an existing bucket, which enables versioning as part of the operation — plus per-object or default retention settings. An object version can carry a retention period, a legal hold, both, or neither. Protection applies to versions rather than keys, which is why versioning is structural rather than incidental. A bucket policy can set minimum and maximum allowable retention periods, which is the guardrail that prevents a mistyped date creating a century-long commitment. Lifecycle expiration cannot delete a version still under retention or a hold, so Object Lock and lifecycle rules compose safely — and should be designed together.

Amazon S3 User Guide — Object Lock

Governance mode

The right choice for operational safety with a documented emergency path, and the documented way to pilot a retention configuration before committing to compliance mode. The bypass requires both the permission and an explicit request header, and every bypass is recorded in CloudTrail — so the small group that holds it is audited rather than trusted. AWS Backup Vault Lock uses the same word for the same idea: manageable by principals with sufficient IAM permissions.

Compliance mode

The retention mode cannot be changed and the retention period cannot be shortened, only extended, which makes a mistyped date an unfixable and billable commitment. It is the answer to any requirement phrased as “must not be deletable even by an administrator” or “including the root user”. Pilot the configuration in governance mode first, and cap the allowable period with a bucket policy. AWS Backup Vault Lock has an equivalent compliance mode that becomes immutable after a grace period, where the contents cannot be altered or deleted by any user or by AWS.

Legal hold

Placed and removed with s3:PutObjectLegalHold, and entirely independent of retention periods — an object version can have a retention period, a legal hold, both, or neither. It is the answer when preservation is required indefinitely with no known end date, typically pending litigation, and must be releasable later. A lifecycle expiration cannot delete a version under a legal hold, which is what makes it a dependable preservation instrument rather than a label.

S3 versioning

The floor of the integrity ladder and a prerequisite for Object Lock, MFA delete and replication. It protects against accidental overwrites and deletions, but a principal with s3:DeleteObjectVersion can permanently remove a specific version — so versioning defends against accidents, not adversaries. In versioned buckets, remember the cost side: noncurrent versions accumulate until a lifecycle rule expires them, and expired delete markers need cleaning up too.

MFA delete

Genuinely strong and genuinely awkward: it cannot be configured in the console, it must be enabled by the root user of the bucket-owning account with MFA, and the protected operations must then be performed with root credentials and a code. That friction is why Object Lock is usually the better answer to a retention requirement. Note the scope: MFA delete protects S3 object versions, not backup vaults and not recovery points.

S3 Glacier Vault Lock

Attach a vault lock policy, validate that it behaves as intended, then complete the lock within 24 hours or abort. Once completed, the policy cannot be changed. The 24-hour window is the pattern worth carrying away: irreversible controls deserve a rehearsal step, which is also why AWS Backup Vault Lock has a cooling-off grace period and why Object Lock can be piloted in governance mode.

AWS Signer and code signing

Integrity applies to what runs, not only to what is stored. A Lambda code-signing configuration names trusted signing profiles and decides whether an expired or revoked signature warns or rejects the deployment; container images get the equivalent through signing and verification in the registry. The exam phrase to recognise is “ensure only approved, unmodified code is deployed”, which is code signing — not a checksum, and not a pipeline approval step.

EBS encryption by default

The answer to “ensure no unencrypted volumes can be created”, and the per-Region scope is the detail that catches people: it must be enabled in every Region you operate in. It applies only to new resources — existing unencrypted volumes cannot be encrypted in place, and the conversion is snapshot, copy the snapshot with a KMS key, create a volume from the encrypted copy, and swap it in. Inheritance runs one way and is absolute: snapshots of encrypted volumes are encrypted, and volumes created from encrypted snapshots are encrypted, so plaintext cannot appear from an encrypted lineage.

RDS encryption at rest

There is no in-place conversion: encrypting an existing unencrypted instance means taking a snapshot, copying it with a KMS key, restoring the encrypted copy to a new instance, and cutting applications over — a sequence with real downtime that the exam expects you to recognise rather than wish away. It is distinct from transparent data encryption, which is an engine feature for specific engines. Prevent the problem with AWS Config rules and with SCPs or RCPs that refuse creation of unencrypted databases.

Transparent data encryption (TDE)

TDE encrypts data inside the database engine; RDS encryption at rest encrypts the underlying storage, snapshots and backups with a KMS key and is invisible to SQL. A requirement that names TDE, or says the encryption must be managed within the database by the DBA team, means the engine feature. They can coexist, but layering them costs performance for limited additional benefit, so AWS advises choosing deliberately instead of enabling both reflexively.

Encrypted snapshot sharing

The rules the exam returns to repeatedly. An aws/rds or aws/ebs key policy cannot be edited, so the only route is to copy the snapshot specifying a customer managed key and share the copy. Sharing then takes two actions: add the account to the snapshot’s restore list, and grant that account use of the KMS key — miss the second and the target sees the snapshot but cannot restore it. The receiving account should then copy it onto a key of its own, or its restored resources stay dependent on a key you could disable. Encrypted snapshots can never be made public; only unencrypted ones can, which has been a real breach vector.

Secret rotation

Secrets Manager invokes the rotation function once per step with the secret id, a client request token and the step name. The design puts promotion last on purpose: the new value is created and stored as AWSPENDING, applied to the target service, tested, and only then promoted by moving AWSCURRENT — so a credential that does not work never becomes the current one. For supported databases AWS offers managed rotation, where you operate no function at all. For zero-downtime rotation on busy databases, the alternating-users strategy keeps two users and rotates them in turn, so the credential being changed is never the one in use.

AWS Secrets Manager User Guide — rotating secrets

Staging label (AWSCURRENT, AWSPENDING, AWSPREVIOUS)

The mechanism that makes rotation invisible to applications: GetSecretValue with no version specified returns AWSCURRENT, so the cutover is a label move rather than a reconfiguration. Retaining AWSPREVIOUS gives a client with a slightly stale cached value a recovery path, and a rollback path for a bad rotation. Two diagnoses follow from the labels: a lingering AWSPENDING version with the old credential still working means rotation failed before finishSecret, usually at setSecret; applications failing right after a successful rotation usually means caching AWSCURRENT too long with no retry on an authentication error.

Rotation function

Its four steps are a contract: createSecret generates the value and stores it as AWSPENDING, setSecret changes the credential in the target service, testSecret verifies the pending value works, finishSecret moves AWSCURRENT. Most rotation failures are setSecret failures — the function lacks privileges to alter the credential, or it sits in a private subnet with no path to the database or to the Secrets Manager endpoint. AWS publishes templates for common targets, and managed rotation removes the function entirely for supported databases.

Parameter Store SecureString

Retrieved with GetParameter and WithDecryption, authorized by IAM plus the key policy. Standard parameters are free and advanced parameters are inexpensive, with no per-secret monthly charge. What it lacks decides the selection rule: no built-in rotation, no staging labels, no per-parameter resource policy (cross-account sharing goes through AWS RAM for advanced parameters), and no cross-Region replication of values. Use it for configuration, endpoints, AMI ids and feature flags; use Secrets Manager for anything that must rotate or be shared cross-account by resource policy.

Amazon SNS message data protection policy

Direction plus operation is the combination to understand: an inbound statement evaluates publishes, an outbound statement evaluates deliveries to specific subscriptions, and each can be scoped to accounts or principals. So you can accept a payload internally while stripping personal fields before they reach an external subscriber, or refuse the publish outright. Detection uses the same managed and custom data identifiers as CloudWatch Logs and Macie, and audit findings go to a log group, a bucket or a Firehose stream. The capability logs lack is deny — SNS can refuse, masking in logs cannot.

Amazon SNS Developer Guide — message data protection

AWS Backup

A backup plan declares what to protect, how often, how long to retain, which vault to write to, and which cross-account or cross-Region copies to make. Vaults have their own resource policies and KMS keys, which is what lets a copy in a separate account be write-only from production. Restore testing performs and validates restores on a schedule, converting “we have backups” into “we have restores” — the only claim that survives an incident. At organization scale, backup policies in AWS Organizations push plans to member accounts so a new account cannot silently be unprotected.

AWS Backup Vault Lock

The control that answers “backups must survive a compromised administrator”. In compliance mode, once the cooling-off grace period you set expires, the lock cannot be removed and recovery points cannot be deleted before their retention expires — by anyone, including AWS. The lock also enforces minimum and maximum retention independently of the backup plan, which defeats the edit-the-plan-and-wait attack. Use the grace period to validate the configuration, because afterwards you are committed to paying for that retention. Pair it with a separate backup account: the account boundary and the lock together are the documented anti-ransomware pattern.

AWS Backup Developer Guide — Vault Lock

Ransomware-resilient backup design

The threat model is an adversary holding your administrator’s credentials, not a failed disk — so anything they can delete is not a backup. Three attacker alternatives to deletion each have a counter: disabling the KMS key (use a key owned by the backup account, and alarm on DisableKey and ScheduleKeyDeletion), shortening retention (Vault Lock enforces minimum retention independently of the plan), and poisoning the backups by encrypting data quietly until every recent recovery point is worthless (retention deeper than the dwell time, restore testing, and detection that shortens dwell time). Immutability without detection just guarantees a long archive of unusable data.

Amazon Data Lifecycle Manager

The EBS-specific, lighter-weight alternative to AWS Backup: policies target volumes or instances by tag and define schedules with retention by count or age. Choose it when the requirement is narrowly about EBS snapshots and AMIs; choose AWS Backup when one plan must cover several resource types, or when vault locking, cross-account vaults and restore testing are needed.

AWS DataSync

The mechanism for migrations and for scheduled replication of file data — into S3, EFS, FSx and between AWS storage services — with built-in integrity verification and bandwidth controls. It is not a backup product: it moves data rather than managing recovery points, retention or restores, which is why a question about immutable backups wants AWS Backup and a question about getting file data into AWS wants DataSync. It is also the usual tool for the EFS and FSx migrations that at-rest encryption changes require, since those file systems cannot be converted in place.

EFS lifecycle policy

Storage-class and cost management rather than retention: there is no expiration or deletion action in an EFS lifecycle policy, so deleting EFS data is an application or AWS Backup concern. It sits beside two neighbours it is routinely confused with, so hold the contrast: S3 Lifecycle both transitions and expires, EFS lifecycle only transitions, and an FSx for Lustre backup policy governs automatic daily backups and their retention.

AWS Organizations

An organization has one management account (the payer, and the only account that can create it), a root at the top of the hierarchy, organizational units nesting up to five levels, and member accounts each in exactly one OU. It runs in all features mode (policies, delegated administration, every integration) or consolidated billing only (one bill, no policy control) — almost everything in SCS-C03 Domain 6 requires all features. Accounts are created from the organization or invited into it, and integrations work through trusted access plus a delegated administrator. Treat the OU tree as the unit of differentiated policy: it is what lets production forbid exactly what development permits.

Management account

The management account creates the organization, attaches policies, designates delegated administrators and consolidates billing. Three properties make it dangerous to use for anything more. Service control policies and resource control policies do not apply to it, so workloads there run outside your own guardrails. It cannot be moved into a quarantine OU or closed without dismantling the organization. And it holds the credential that can undo everything. The exam’s remediation is always the same: move capability out of it — enable trusted access, designate a delegated administrator in a dedicated security account, and leave the management account doing billing and policy.

Organizational unit (OU)

OUs nest up to five levels below the root, and an account belongs to exactly one at a time. Nearly every mechanism in this domain targets an OU rather than an account list: SCPs and RCPs, declarative and tag policies, Control Tower controls, service-managed StackSets with automatic deployment, Firewall Manager policy scope, organization Config rules. That is the design goal — a control becomes a property of being in this OU, so an account vended next month inherits it with no human action. The containment corollary: moving a compromised account into a quarantine OU whose SCP denies nearly everything freezes it in seconds without destroying evidence.

All features mode

Service control policies, resource control policies, declarative policies, tag policies, AI services opt-out policies, delegated administrators, service-managed StackSets and AWS Firewall Manager all require all features. If a policy type will not enable or a service will not offer organization-wide configuration, the organization’s mode is the first thing to check. An organization created as consolidated-billing-only can be switched to all features (member accounts are asked to approve), without being recreated.

Trusted access

Enabled in the management account, per service. Without it, "enable this for the whole organization" is simply not offered, and a delegated administrator cannot be designated. It creates the service-linked roles the integration needs in member accounts. The exam reflex for any "centrally manage X across all accounts" stem is a two-step: enable trusted access for X, then designate a delegated administrator in the security tooling account — followed by the modifier questions, is this per Region, and does the requirement need enabling (organization rules, conformance packs, configuration policies) rather than just viewing (an aggregator).

OrganizationAccountAccessRole

An account created with CreateAccount (or through Control Tower Account Factory) is born with this role, letting the management account administer it without a root password. An account that was invited keeps its existing configuration and gets no such role, which is the answer to “we invited the account and now cannot administer it” — create the cross-account role yourself, or onboard the account into IAM Identity Center. Either way, long-term credentials in member accounts are not the answer.

Security tooling account

Conventionally in a Security OU, this account holds the delegated administrator role for Security Hub, GuardDuty, AWS Config, Inspector, Macie, Detective, IAM Access Analyzer, Audit Manager and Security Lake, plus the roles responders assume into workload accounts. Why not the management account, which could do all of it? Blast radius, least privilege, and the fact that SCPs and RCPs do not constrain the management account — so your security team would operate outside your own guardrails. AWS recommends keeping the management account out of administrator roles. It is a control plane, not a tenant: no application workloads belong here.

Log archive account

Logs land where the principals that generated them cannot administer them. Add S3 Object Lock, a bucket policy denying deletes, a separate KMS key, and read-only human access, and compromising a workload account no longer edits the audit record of the compromise. The exam phrases it as “logs must survive compromise of the account that produced them”, “immutable”, or “tamper-proof”. Control Tower creates this account as part of the landing zone; built by hand it is an organization trail plus a Config delivery channel aimed at one central bucket.

AWS Control Tower

Control Tower builds and then maintains a multi-account baseline. Every piece is something you could assemble from Organizations, CloudFormation and AWS Config by hand; what it adds is continuity — the baseline is re-applied to every account it vends, divergence is reported as drift, and controls arrive as a curated catalogue instead of policy JSON you maintain. It supports new and existing environments: enrol accounts individually or register a whole OU, clearing conflicts first (a pre-existing Config recorder in the Region is the classic blocker). Extend it with Customizations for AWS Control Tower (CfCT) rather than by hand-editing what it owns.

Landing zone

The artifact Control Tower sets up and keeps aligned: a Security OU containing the log archive and security tooling accounts, an organization CloudTrail trail delivering to that archive, AWS Config recorders and an aggregator, IAM Identity Center for human access, Account Factory for vending, and mandatory and recommended controls attached to the OUs. The concept predates the service — a hand-built landing zone is a legitimate architecture, and the trade is that you own the drift detection and the account baseline yourself.

Control (formerly guardrail)

The mechanism matters because each kind fails differently. Preventive controls are implemented as service control policies: the API call returns AccessDenied and nothing is created, so there is no finding to triage. Detective controls are AWS Config rules: the resource exists and is reported NON_COMPLIANT. Proactive controls are CloudFormation hooks running CloudFormation Guard rules: the stack operation fails before provisioning. Guidance levels are mandatory (always on, protecting the landing zone’s own machinery), strongly recommended (AWS best practice, switchable) and elective (opt-in). If you cannot name which kind a control is, you cannot predict how it fails — and that is what the exam asks.

Proactive control

The third control kind, and the newest, so candidates still reach for a Config rule out of habit. A detective control reports the unencrypted volume; a preventive control denies the API; a proactive control fails the CloudFormation stack, so the resource never comes into existence at all. The limit is its scope: it only sees resources created through CloudFormation. A console click, a CLI call or a Terraform apply walks straight past it — which is why the honest architecture pairs it with a preventive control (deny the API whatever the caller) and a detective control (catch everything else, including what existed beforehand). “Non-compliant resources must never be deployed in the first place”, in a CloudFormation shop, is the proactive signature.

Account Factory

Teams request an account rather than filing a ticket to the management account, and every account arrives identical: the Control Tower baseline, the network configuration you templated, IAM Identity Center access already wired, and OU membership that makes every OU-scoped control automatic. Standardization is the control — the accounts nobody standardized are the ones that fail an audit. Account Factory is literally a Service Catalog product in the management account’s portfolio, which is why vending an account looks like launching one; Account Factory for Terraform offers the same lifecycle for Terraform shops.

Drift (landing zone and account)

Someone moved an account between OUs, detached a mandatory SCP, deleted a Config recorder, modified the organization trail. Control Tower detects these and offers to repair the landing zone. Two consequences for the exam. Hand-editing the policies Control Tower owns (their names begin aws-guardrails-) does not disable a control — it creates drift. And the supported way to add your own policy is your own separate SCPs or Customizations for AWS Control Tower, not editing what the service manages. Note this is a different thing from CloudFormation stack drift, which compares deployed resources against a template.

Declarative policy

An API deny list protects today’s API surface; a new API or console path can route around it until someone updates the policy. A declarative policy instead states the intended configuration — public access to AMIs and snapshots blocked, only approved AMI owners allowed, instance metadata defaulting to IMDSv2, serial console access off, EBS encryption by default — and AWS enforces that intent as the service evolves, reporting the enforced state per account. Attach it to the root, an OU or an account; it needs all-features mode and the policy type enabled on the root. As of October 2026 support centres on Amazon EC2 and its adjacent VPC and EBS settings. The exam phrasing to recognise: “must remain enforced even as new service features are released”.

AI services opt-out policy

A data-governance consent setting rather than a permission or configuration control. Attach it at the root and every covered account is opted out, instead of chasing a per-service console setting in each account or opening a support case. It requires all-features mode and the policy type to be enabled on the organization root. On the exam this is the easy mark in the policy-types question: a requirement that content processed by AWS AI services must not be retained for service improvement is one policy, attached once.

Tag policy

A tag policy fixes environment versus Environment and restricts values to an approved set, reports non-compliant tagging, and can enforce the shape for supported resource types. What it cannot do is require a tag to be present — and that is the trap. “Every resource must carry a CostCenter tag at creation” is an IAM or SCP Deny conditioned on aws:RequestTag/CostCenter being absent (with aws:TagKeys for the key set). Use both: the deny makes tagging mandatory, the tag policy makes it consistent. Tags then drive Firewall Manager scope, Config rule scope, Backup selections, Service Catalog TagOptions, cost allocation, and attribute-based access control.

Allow-list strategy (SCP)

FullAWSAccess is attached by default to the root, every OU and every account, so out of the box you are running a deny-list model: everything is permitted except what a Deny removes. Detach it from a target and that subtree flips to an allow-list: only what you explicitly allow is reachable, and future AWS services are excluded automatically with no list to maintain. Powerful, and the fastest way to break an organization before lunch — which is also why “I attached an Allow-only SCP and nothing changed” is almost always FullAWSAccess still being attached alongside it.

Policy inheritance (effective permissions)

There is no override, no precedence and no “closest wins”: a permission must survive every level of the tree. If the root’s SCP omits dynamodb:*, nothing below the root can use DynamoDB, whatever is attached lower down. Multiple policies at the same target combine, which is why FullAWSAccess sitting beside your narrow Allow keeps the ceiling wide open. RCPs inherit the same way, over resources instead of principals. Final layer: the effective permission on a call is the intersection of the SCP chain, the identity policy, any permission boundary and session policy, the resource policy, and the RCP chain — all of which must allow.

Break-glass procedure

The design points the exam cares about: the trigger is defined in advance, not at 3am; the credential is split between custodians (password envelope and hardware MFA held separately) so nobody acts alone; use fires an alarm in real time, which is what distinguishes an approved break-glass from an attacker who reached the same credential; the session performs only the pre-approved task; and afterwards everything is rotated, re-sealed and written up. Prefer a dedicated break-glass IAM role per account over the root user, and prefer a centrally assumed, expiring root session over a stored root password. Standard detection: a CloudWatch Logs metric filter over CloudTrail matching $.userIdentity.type = "Root" (excluding invokedBy and AwsServiceEvent) with an alarm at a threshold of one, plus an EventBridge rule on root ConsoleLogin.

CloudFormation StackSet

Define the template and parameters once in an administrator account and target accounts × Regions; CloudFormation creates and tracks a stack instance in each combination, and every instance reports its own status (SUCCEEDED, FAILED, OUTDATED) back to the stack set — which is how one view answers where is this control missing right now. Operation preferences bound the rollout itself: maximum concurrent accounts, failure tolerance, and sequential or parallel Regions. Drift detection is an operation you run, reporting instances whose resources no longer match the template. And when an account leaves a target OU, the account removal behaviour decides whether its stacks are deleted or retained — a choice worth making deliberately.

Service-managed permissions (StackSets)

Requires AWS Organizations with all features plus trusted access for StackSets, and runs from the management account or a registered delegated administrator. Targets are OUs or the whole organization and CloudFormation uses service-linked roles, so there is nothing to deploy into a target account first. The alternative, self-managed permissions, targets a list of account ids and needs an administration role in the admin account plus a matching execution role in every target — which is what you use for accounts outside your organization, such as an acquisition that has not joined yet. Exam tell: “all accounts, including those created in the future” is service-managed with automatic deployment; “a partner’s account” is self-managed.

AWS CloudFormation Guard

Guard answers is this allowed by our policy? — encryption declared, no public access, approved AMIs, required tags. The reason it matters for SCS-C03 is reuse: the same rule set runs as a CLI gate in your pipeline, behind AWS Config custom policy rules (a Config rule with no Lambda function to host), and as the engine of Control Tower proactive controls implemented as CloudFormation hooks. Contrast cfn-lint, which checks whether a template is valid rather than whether it is permitted. Beyond CloudFormation it can evaluate other structured configuration, including Terraform plans rendered as JSON.

cfn-lint

It catches the deploy that would have failed at 70% rollout: a property that does not exist on that resource type, a required property omitted, an intrinsic function used wrongly, a value outside an allowed enumeration, a resource type unavailable in the target Region. It says nothing about whether your organization permits what the template describes — that is CloudFormation Guard. A mature pipeline runs lint first, because policy-checking a template that cannot deploy is wasted work.

AWS Firewall Manager

Define a policy once in a designated administrator account, scope it to accounts, OUs, resource types and tags, and Firewall Manager keeps in-scope resources configured — including resources and accounts that appear later, and including resources somebody deliberately detached. Prerequisites are exam material: AWS Organizations with all features, a designated Firewall Manager administrator account (several are supported, each scoped to particular OUs, Regions and policy types), and AWS Config enabled in every account and Region a policy covers — a Region with Config off is a Region where policies find nothing. Shield Advanced policies also need the subscription; Network Firewall and DNS Firewall policies need RAM sharing enabled in the organization.

Firewall Manager policy

Each policy chooses its posture: automatic remediation, where Firewall Manager brings non-compliant resources into line itself, or audit only, where it reports and leaves them alone. On a brownfield estate start audit-only, read the compliance report, fix the applications the policy would have changed, then switch remediation on — enabling it blind across dozens of accounts is how a rollout becomes an outage. The three security group variants are worth keeping straight: common replicates and associates a primary security group, content audit checks existing rules against allowed or denied templates (and can strip offending rules), usage audit finds unused and redundant groups. Third-party vendor firewalls are managed through the same framework.

Organization Config rule

The distinction that catches people: a Config aggregator reads compliance results that already exist, so an aggregator over accounts with no recorder or no rules faithfully reports nothing — which renders as clean. Organization rules and organization conformance packs are what actually switch evaluation on in members, including accounts created later. Deploy them from the AWS Config delegated administrator in the security tooling account, exclude named accounts where a genuine exception exists, and pair with the aggregator for the single cross-account view and its SQL advanced queries.

AWS Resource Access Manager (AWS RAM)

Enable sharing with AWS Organizations once and shares to principals inside the organization are accepted automatically; a share to an account outside the organization still needs an invitation and acceptance. Commonly shared: VPC subnets (central networking owns the VPC, applications deploy into it), Transit Gateways, AWS Private CA, License Manager configurations, Route 53 Resolver rules and DNS Firewall rule groups, Network Firewall rule groups and policies, managed prefix lists, EC2 Capacity Reservations and Dedicated Hosts, Image Builder components, and Lake Formation-governed Glue objects. Not shareable: S3 buckets, SQS queues, SNS topics, KMS keys and Lambda functions — those use their own resource policies, and “use RAM to share an S3 bucket” is a reliable wrong answer.

Resource share

Be precise about what a share does not delegate. Sharing a subnet does not make the participant a VPC administrator: participants create resources in the shared subnet but cannot modify the VPC, its subnets, route tables or network ACLs, and cannot see or touch resources belonging to other participants; the owner cannot modify participant-created resources either. Sharing a private CA lets a participant issue certificates, not delete the CA. And a share never grants IAM permissions — participants still need identity policies allowing their actions, which is why “we shared it and they still get AccessDenied” is an IAM question. Two controls sit on top: create shares with external principals disallowed (deny the permissive variant organization-wide with an SCP condition on ram:RequestedAllowsExternalPrincipals), and run IAM Access Analyzer with the organization as its zone of trust to inventory what actually leaves.

AWS Service Catalog

A portfolio holds products the platform team wrote and reviewed; constraints shape how each may be launched. Launch constraints name the role Service Catalog assumes to provision, so end users need permission to launch the product and nothing else. Template constraints restrict parameter values (instance types, allowed CIDRs) so a product cannot be launched into a non-compliant shape. Notification constraints publish stack events to SNS; stack set constraints let one launch deploy across accounts and Regions; tag update and resource update constraints govern later changes; TagOptions push required tags onto every provisioned product. Portfolios are shared with accounts, an OU or the organization — and Control Tower Account Factory is itself a Service Catalog product.

Launch constraint

This inverts the usual trade. Normally “let developers deploy their own environments” means granting broad permissions and hoping a Config rule catches the mistakes; with a launch constraint the permissions live in the launch role and are usable only through a template you authored and versioned. The only infrastructure they can create is infrastructure you already approved, and the audit trail shows which product version produced which stack. Exam tell: developers must not have permissions to create the resources directly is a launch-constraint question every time.

Config remediation action

Detection that fixes nothing is a report, not a control. A remediation action names a runbook (AWS-EnableS3BucketEncryption, AWS-DisablePublicAccessForSecurityGroup, or your own), its parameters, and the role it assumes; automatic remediation fires on the verdict and retries up to a configured number of attempts within a retry period before stopping. The loop closes by itself: the runbook’s change produces a new configuration item, the rule re-evaluates it, and COMPLIANT appears — that second evaluation is the evidence the fix worked. Choose manual remediation (or scope exceptions by tag) when the fix could break a live workload, when legitimate exceptions exist, or when the remediation role would need permissions wide enough to become an escalation path.

AWS Audit Manager

You create an assessment from a framework, scoped to the accounts and services in question, and Audit Manager accumulates evidence from then on, so “show me that this control operated in March” is a navigation rather than an investigation. A control set can be delegated to a subject-matter expert who reviews, comments and attaches evidence without becoming an administrator, and an assessment report (report plus evidence files, with validation) is generated into an S3 bucket you own. In an organization, designate a delegated administrator and one assessment spans member accounts. The exam’s pairing: Artifact gives you AWS’s evidence about AWS; Audit Manager gathers yours about you. Neither makes you compliant — they evidence it.

Audit Manager framework

AWS ships frameworks for the common standards — CIS, PCI DSS, SOC 2, HIPAA, GDPR, NIST 800-53, ISO 27001, and a generative-AI best-practices framework — and a custom framework lets you encode a control catalogue that does not map one-to-one onto any published standard, which is what most regulated organizations actually need. Each control names its data sources (AWS Config, Security Hub, CloudTrail, API calls) or is marked manual. A framework is a template; an assessment is the running instance of it.

Audit Manager assessment

Create it once and it works continuously, which is the property that answers an auditor’s real question. Audits are about a period, not an instant: a Security Hub dashboard showing 100% compliance today says nothing about March, whereas an assessment running across the year holds dated, per-control evidence for every month of it. Control sets can be delegated for review, manual evidence attached, and the final assessment report generated into your own S3 bucket with validation so the recipient can confirm nothing was altered.

Evidence (Audit Manager)

Four kinds, and the exam distinguishes them. Compliance check — results from AWS Config rules and Security Hub controls, which is why your Config estate is your evidence pipeline and a weak one produces a weak audit. User activity — relevant CloudTrail events, and the reason log retention and integrity validation are audit requirements rather than preferences. Configuration data — resource snapshots gathered through describe and list API calls. Manual evidence — the signed policy, the training completion record, a vendor attestation, a penetration test report: things no API produces, and without which an assessment is incomplete. Evidence lands in folders organised by date and control.

AWS Artifact

Two halves that get confused. Reports: AWS’s own audit artifacts (SOC 1/2/3, ISO 27001 and its siblings, PCI DSS AOC and responsibility summary, FedRAMP, C5, IRAP), usually under NDA, plus third-party reports for some products sold through AWS Marketplace. Agreements: contracts you accept, such as a Business Associate Addendum for HIPAA workloads — and an organization agreement is accepted once in the management account on behalf of every member account, which is the right answer to “we must accept the BAA for all 60 accounts”. Artifact is how you evidence inherited controls: it will never tell you whether your bucket is encrypted, but it proves the facility holding the disk is audited.

AWS Well-Architected Tool

The only instrument in this domain that evaluates design rather than resource state, because humans answer it: how is least privilege established, how would a compromised credential be contained, what is the data classification, how is the pipeline validated. You define a workload (name, environment, Regions, account ids), apply lenses, work through the questions, and each unimplemented best practice becomes a risk — the serious ones classified as high-risk issues with prescriptive guidance in an improvement plan. Save a milestone and the next review shows the delta instead of starting over. Workloads and custom lenses can be shared with accounts, OUs or the organization, and the tool can surface relevant Trusted Advisor checks inside a review.

Lens (Well-Architected)

The base lens is the AWS Well-Architected Framework with its six pillars. The lens catalog adds domain-specific sets (serverless, SaaS, machine learning, data analytics, IoT, financial services, generative AI). A custom lens lets you express your own security standard as questions and best practices and share it with other accounts, an OU or the whole organization — so every product team is reviewed against the same bar, and milestones make each team’s progress comparable between reviews. The Security Pillar itself walks the ground this certification covers: identity and access management, detection, infrastructure protection, data protection, incident response, application security.

High-risk issue (HRI)

Reviews produce high-risk and medium-risk issues rather than a pass or fail, and the improvement plan is the deliverable — an ordered list of what to change, with AWS guidance attached. Because milestones snapshot a review, an HRI closed between reviews is visible as progress, which is what turns an annual ritual into a programme. An HRI is the kind of finding automation was never going to surface: “no documented way to isolate a compromised instance” has no configuration item and no control id.

AWS Trusted Advisor

The security checks to recognise: unrestricted security group ports, S3 bucket permissions open to public or authenticated users, IAM use, MFA on the root user, exposed access keys, public RDS and EBS snapshots, CloudTrail logging not enabled, load balancer listener security. The overlap with Security Hub controls is real; the difference is consumption — Security Hub evaluates every resource continuously, scores it against a named standard and feeds workflows, while Trusted Advisor is a periodic advisory read of an account. Support plan decides availability: a core subset of security checks and the service quota checks are available on every plan, while the full set, programmatic access through the AWS Support and Trusted Advisor APIs, refresh, notifications and the organizational view need Business, Enterprise On-Ramp or Enterprise support.

DEA-C01

The only version of this exam — there is no C02. You get 65 questions, of which 50 affect your score and 15 are unidentified pilot items; 130 minutes; a scaled score of 100–1,000 with a minimum passing score of 720. Scoring is compensatory, so there is no per-domain pass bar. Two response types only: multiple choice (one correct of four) and multiple response (two or more correct out of five or more). Four content domains, weighted by share of scored content: Data Ingestion and Transformation 34%, Data Store Management 26%, Data Operations and Support 22%, Data Security and Governance 18%. The current exam guide revision is 1.1, published 12 December 2025, which merged the old separate "knowledge of" and "skills in" lists into one list per topic and added content on large language models in data processing, open table formats, vector index types, business catalogs, data sovereignty and data-sharing patterns.

AWS Certified Data Engineer – Associate exam guide, revision 1.1 (December 2025)

Target candidate (DEA-C01)

The published description is specific: roughly 2–3 years of data engineering experience, an understanding of how volume, variety and velocity affect ingestion, transformation, modeling, security, governance, privacy, schema design and store choice, and at least 1–2 years of hands-on AWS experience. Expected general IT knowledge includes building and maintaining extract-transform-load pipelines end to end, language-agnostic programming concepts, Git for source control, using data lakes, general networking/storage/compute, and general concepts of vectors. Expected AWS knowledge includes comparing services on cost, performance and function, writing and running SQL on AWS services, and verifying data quality and consistency. Three job tasks are explicitly out of scope: performing machine learning training and inference, demonstrating language-specific syntax knowledge, and drawing business conclusions from data. That last list is a practical filter — an option that only makes sense if you are training a model or interpreting a business result is almost always a distractor.

AWS Certified Data Engineer – Associate exam guide, target candidate description

Pacing (two minutes per question)

The arithmetic is unusually clean on this exam: 130 minutes over 65 questions is 2 minutes 0 seconds per item, which makes checkpoints easy to hold in your head — question 20 by minute 40, question 40 by minute 80, all 65 touched by minute 110, leaving 20 minutes for flagged items. Run the first pass faster than average so the second pass has somewhere to live. Two honest complications. Multiple-response items with five or six options read slower than two minutes, so they spend the slack that the easy recognition items create. And you cannot tell which 15 items are unscored, so an item that feels unfairly obscure still gets full effort — it may simply never have counted. Unanswered questions are scored as incorrect and there is no penalty for guessing, so a blank is strictly worse than any guess.

Deciding constraint

Most well-built data-engineering items offer three or four options that would all work in some pipeline, and exactly one clause in the stem that makes three of them wrong. On this exam the deciding constraint almost always belongs to one of five families: latency ("within seconds", "nightly", "sub-millisecond"), volume and throughput ("petabyte", "two million records per second", "a 40 GB file"), query shape ("ad hoc over three years of history", "look up one customer by id", "join eight dimensions"), cost and operational effort ("minimum operational overhead", "no clusters to manage", "cheapest at rest"), and governance ("must not leave the Region", "column-level access by team", "retain seven years, provably unaltered"). The discipline is to name the family out loud, then ask of each option whether it satisfies that clause — rather than asking which option is the best pipeline, which is a question with several defensible answers.

Keyword reflex

Reflexes buy reading time: "no servers to manage" shortlists serverless services, "exactly once into the warehouse" shortlists an idempotent load keyed on a business key, "sub-millisecond key lookup" shortlists an in-memory store. They are worth drilling because they cut four options to two in seconds. They are dangerous for exactly the same reason: a well-built item plants the keyword and then adds the clause that breaks it, so "no servers to manage" sits in a stem that also demands a specific Hadoop ecosystem tool, and the serverless answer loses. The rule: let a reflex shortlist, then let the deciding constraint choose. A reflex that survives the constraint check is an answer; a reflex you never checked is a guess that felt like knowledge.

Bounded data

Bounded data is finite and complete at the moment you read it, which is what makes batch processing possible: a job can see every row, so totals, joins, sorts and deduplication are exact. The boundary is usually a time window or a file manifest. Bounded does not mean small — a 40 TB daily export is bounded. It means you can answer "is that all of it?" with yes. The exam uses the distinction as a silent constraint: a stem describing a nightly file drop has given you bounded data and ruled out the stream processors, and a stem describing continuous clickstream events has done the opposite.

Unbounded data

Unbounded data never completes, so there is no moment at which a job can read all of it. Processing therefore happens over windows — the last five minutes, a tumbling hour, a session — and every such answer is provisional, because a record delayed in the network can arrive after its window closed. That single fact drives most of streaming’s complexity: watermarks, allowed lateness, idempotent sinks and replay all exist because the input has no end. Two consequences the exam leans on: an unbounded source needs a consumer that is always on, and a correction to an already-published window means reprocessing, not editing.

At-least-once delivery

A producer or consumer that cannot confirm an acknowledgement retries, and a retry after a successful-but-unacknowledged write produces a duplicate. At-least-once is therefore the normal guarantee: nothing is lost, some things are doubled. Its sibling at-most-once drops rather than duplicates, which is almost never acceptable for billing or audit data. The engineering answer is not to chase perfect delivery but to make the destination idempotent, so a duplicate is harmless. On the exam, "duplicate rows appeared in the warehouse after a retry" is not a bug report about the stream — it is a question about the sink.

Exactly-once processing

Exactly-once is a property of the whole path, not of a transport. Transport layers can offer exactly-once within their own boundary — a stream framework can checkpoint offsets atomically with its state — but the moment the data leaves for an external store, the guarantee has to be re-established there: a deduplicating primary key, a conditional write, a transactional commit, or a merge keyed on a business identifier. The practical form of the sentence is "at-least-once delivery into an idempotent sink", and it is worth being able to say it that way, because an option promising exactly-once with no mention of the destination is describing something that does not exist.

Idempotency

An idempotent write can be repeated without changing the outcome: an upsert on a natural key, a conditional put that fails if the item exists, writing to a deterministic object key so a rerun overwrites rather than appends, or a merge that matches on a business identifier. A non-idempotent write — an unconditional insert, an append, a counter increment — turns every retry into corruption. Idempotency is the cheapest insurance in a pipeline because it makes three other things free: automatic retries stop being dangerous, replay from a stream becomes a recovery tool rather than a risk, and a backfill can be rerun after it fails halfway.

Replayability

Replay is the difference between a bad deployment being an inconvenience and being permanent data loss. A source is replayable when it keeps the records for a retention period and lets a consumer restart from an earlier position — so you fix the transformation, reset the position, and the corrected results flow through again. It is not replayable when the records are forwarded and forgotten, or when the only copy was the one your broken job just wrote. The design move is to land raw records immutably before transforming them, which buys replayability even from a source that has none. On the exam, "we need to reprocess the last three days after finding a logic error" is a replayability requirement, and it rules out any path with no durable raw copy.

Event time vs processing time

Event time is the timestamp carried in the record — when the sale rang up, when the sensor read. Processing time is the clock on the machine that handled it. They diverge whenever anything is slow, retried, buffered on a mobile device, or replayed, and the gap can be seconds or days. Aggregating by processing time is easy and quietly wrong: a device that uploads yesterday’s readings this morning lands them in today’s totals. Aggregating by event time is correct and costs you the hard problem — deciding how long to wait for stragglers before you publish a window, which is what a watermark encodes. A replay makes the distinction unmissable: reprocessing last week’s stream today gives identical event-time results and completely different processing-time ones.

Row-oriented storage

A row-oriented file or table keeps all of a record’s fields adjacent, so appending a record is one write and fetching a record is one read. That is exactly what a transactional application wants. The cost appears when an analytical query asks for two columns out of forty: the engine must still travel through all forty, because the two it wants are interleaved with the rest. Row layouts also compress poorly, since neighbouring bytes come from different types. Comma-separated and JSON Lines files are row-oriented, which is why they are the usual landing format and the wrong long-term query format.

Columnar storage

A columnar format stores a block of customer ids, then a block of amounts, then a block of timestamps. Two things follow. First, projection becomes physical: selecting two columns of forty reads roughly two fortieths of the bytes, because the other thirty-eight are never touched. Second, compression works, because a run of values of one type is highly redundant — dictionary and run-length encodings routinely cut a column several-fold. Columnar formats also carry per-block statistics such as minimum and maximum values, which lets an engine skip whole blocks that cannot satisfy a filter. Together those three effects are why converting landed files to a columnar format is the single highest-leverage change to a lake’s query cost.

Bytes scanned

On a pay-per-query engine over object storage, the bill is a rate times the bytes the engine had to read, so the whole craft of cheap analytics is reducing that number without changing the answer. Four levers, in rough order of effect: partition so that a filtered query opens only the relevant prefixes; use a columnar format so that a projected query reads only the named columns; compress, because the bytes counted are the compressed bytes on storage; and keep files large enough that per-file overhead does not dominate. The same levers shorten the query, which is why "it costs too much" and "it takes too long" usually have one fix. A query that selects every column with no filter defeats all four at once.

Predicate pushdown

A naive engine reads everything and applies the filter in memory; a pushdown-capable one hands the filter to the reader, which consults per-block statistics and skips any block whose minimum and maximum values prove it cannot match. The saving is physical, not computational — the bytes are never fetched. Pushdown needs three things to work: a format that carries statistics, a filter the reader can evaluate (a comparison on a stored column), and data laid out so that blocks are not all equally likely to match. That last condition is why sorting or clustering a table on its most-filtered column multiplies the benefit, and why a filter wrapped in a function the reader cannot interpret silently turns pushdown off.

Partitioning

Partitioning encodes a column’s value into the storage path, so a query filtered on that column can eliminate every other path before opening a single file. Choosing the partition column is the whole decision: it must be something queries actually filter on, and it must have low enough cardinality that each partition holds substantial data. Partitioning by date usually works; partitioning by customer identifier usually does not, because it produces millions of tiny partitions whose listing and per-file overhead costs more than the scan it saved. Note the asymmetry that catches people out — partitioning helps only the queries that filter on the partition column, and a partitioned table whose partitions are not registered in the catalog is read as if those partitions did not exist.

The small-files problem

Every file in object storage costs a request to list and a request to open, and a distributed reader cannot usefully split a file smaller than one read block. So a table of 100,000 one-kilobyte files behaves far worse than the same 100 MB in a handful of files, and the gap grows with parallelism because the overhead is per file rather than per byte. Streaming delivery is the usual culprit: a delivery stream flushing every 60 seconds writes 1,440 objects a day per partition. The standard remedies are buffering longer before writing, a scheduled compaction job that rewrites small files into large ones, and avoiding partition columns so fine that each partition can only ever hold a trickle.

Cardinality

A boolean flag has cardinality two; a customer identifier in a consumer business has cardinality in the millions. The number matters because almost every physical design choice is a bet on it. High cardinality is what you want in a key you look rows up by — it spreads load and makes each lookup selective. Low-to-moderate cardinality is what you want in a partition column, so each partition is large enough to be worth opening. Get them the wrong way round and the failure modes are opposite and equally bad: a low-cardinality lookup key creates a hot partition, and a high-cardinality partition column creates millions of tiny files.

Hot partition (hot key)

Distributed stores and streams spread work by hashing or ranging a key. If the key is skewed — a tenant identifier where one tenant is a tenth of the traffic, a date key where every writer writes today, a status column where almost every row says "active" — then one partition receives traffic the others do not, and that partition hits its own limit long before the table or stream hits its aggregate limit. The symptom is throttling at a fraction of provisioned capacity, which looks like a quota problem and is really a key-design problem. The fixes all spread the key: add a random or hashed suffix, pick a higher-cardinality key, or write to a composite key that distributes within the hot value.

OLTP (online transaction processing)

Transactional workloads are defined by their shape rather than their size — thousands of concurrent operations, each selecting or updating a handful of rows identified by a key, with strict latency expectations and transactional guarantees. That shape rewards row-oriented storage, indexes on the lookup keys, and normalised schemas that avoid writing the same fact twice. It punishes full scans. The reason a data engineer cares is that analytics has the opposite shape, so running reports against the application database competes for exactly the resources the application needs — which is the original reason warehouses and lakes exist at all.

OLAP (online analytical processing)

Analytical workloads invert every assumption transactional ones make. There are few concurrent queries, each reads millions or billions of rows, each names only a few of the available columns, and seconds-to-minutes is an acceptable answer time. That shape rewards columnar storage, compression, partitioning and massively parallel scanning, and it makes denormalisation sensible — repeating a dimension’s attributes to avoid a join is a trade analytics is happy to make and transactions are not. Keeping the two shapes straight is what turns "which data store?" from a matter of taste into an elimination: the stem tells you the workload shape, and the shape rules out the stores built for the other one.

Data lake

A lake separates storage from compute: the files sit in object storage, a metadata catalog says what the files mean, and any number of engines — a SQL query service, a Spark cluster, a warehouse reading external tables — read the same bytes without copying them. That buys cheap retention, format freedom and multi-engine access. It costs you the guarantees a database provides for free: there is no enforcement that a writer honours the declared schema, no transaction spanning several files unless you add a table format that provides one, and no index beyond what partitioning and file statistics give you. A lake’s quality is therefore a property of its pipelines and its catalog, not of its storage.

Data warehouse

A warehouse owns its storage and its format, which is precisely what lets it be fast and strict. Because it ingested the data through a load step, it can enforce types and constraints, maintain statistics, choose how rows are distributed and sorted across nodes, and plan complex multi-way joins well. Because it owns the storage, getting data in is a deliberate act with a cost, and the same data living in a lake and a warehouse is two copies to keep consistent. The practical split against a lake: warehouses win on repeated, complex, concurrent SQL over curated data; lakes win on cheap retention, open formats, and workloads that are not SQL.

Lakehouse

The lakehouse pattern keeps the files open and in object storage but puts a table format over them that maintains a manifest of which files constitute the current table. That manifest is what buys the database-like properties a bare lake lacks: an atomic commit across many files, row-level update and delete without rewriting a whole partition, schema evolution that readers can follow, and reading the table as it was at an earlier point. The cost is a new maintenance surface — snapshots, manifests and small files accumulate and need expiring and compacting — and an engine that does not understand the format sees only a directory of data files.

Schema-on-read

Schema-on-read stores files as they arrive and applies a structure at query time, from a catalog entry or from the reader’s own inference. The benefit is that ingestion is decoupled from interpretation: a new field appears in the source and nothing breaks, and the same files can be read under two different schemas for two different purposes. The cost is that nothing validated the data on the way in, so a type that changed from integer to string three weeks ago shows up now as a query error or, worse, as silently null columns in a report. Schema-on-read does not remove the schema problem; it moves it from the writer to the reader and delays it until someone is depending on the answer.

Schema-on-write

Schema-on-write declares the structure first and enforces it at load time, which is how a warehouse or a relational table works. A record with the wrong type, a missing required column or an impossible value is rejected or routed aside, so what is inside the table can be trusted. The price is operational: the load becomes a thing that can fail at three in the morning, and a legitimate change at the source needs a migration rather than nothing at all. The mature pipeline is usually both — land raw with schema-on-read for replayability and audit, then conform into a schema-on-write layer that reports are allowed to depend on.

ETL (extract, transform, load)

In extract-transform-load, a processing engine reads from the source, reshapes the data, and writes the finished result into the destination. The destination stays clean and small, which matters when its storage is expensive or its compute is the thing you are protecting. The costs are inflexibility and loss: a new question that needs a column you dropped means re-extracting from the source, and the transformation engine is a separate system to size, schedule and debug. Extract-transform-load is the right shape when the destination is costly, when records must be validated or masked before they land, or when the source cannot be read again.

ELT (extract, load, transform)

In extract-load-transform, raw data is landed in the lake or warehouse and then reshaped by that system’s own compute — SQL in the warehouse, a job over the lake. Because the raw layer is preserved, a transformation bug is fixable by rerunning over data you still have, and a new requirement can reach columns nobody kept deliberately. The trade is that the destination now stores raw data it will never serve directly and runs transformation work alongside queries, so storage and contention both grow, and the discipline of layering — raw, conformed, curated — becomes load-bearing rather than tidy. Cheap object storage is why this became the default shape.

Backfill

Backfills happen for three reasons: a bug corrupted a window of output, a new column or metric must exist for history as well as for new data, or a new consumer needs more history than the pipeline has produced so far. All three are the same engineering problem — running a large, bounded job over a range that normal operation never touches — and the hazards are consistent. The job must be idempotent, or a partial failure leaves duplicates. It must be partitioned by the range being refilled, so progress is resumable and a rerun targets only what failed. It should be rate-limited, because a backfill competes with live traffic for the same quotas and the same hot partitions. And the old output needs a defined fate: overwritten, versioned, or marked superseded.

Freshness (and the latency budget)

Freshness is measured end to end, from the moment an event happens to the moment a consumer can see it, and it is the sum of every stage: source emission interval, transport buffering, processing time, load time, and any cache in front of the consumer. That sum is why a "real-time dashboard" fed by a five-minute buffered delivery stream and an hourly load is not real-time, however streaming the first hop was. Two things to notice in a stem. A freshness requirement expressed in seconds forces a continuous path all the way through; one expressed in hours makes a scheduled batch both sufficient and much cheaper. And requirements are frequently stated about the consumer ("the dashboard must reflect sales within two minutes"), which is the whole budget, not the budget for one hop.

Data skew

A distributed engine splits work by a key, and if the key values are unevenly distributed then one task receives far more rows than its peers. The job cannot finish until that task does, so adding workers changes nothing — the symptom is 199 tasks finishing in two minutes and one running for forty. Skew is most visible around joins and group-by operations, where all rows sharing a key must meet in one place, and its commonest causes are a dominant value, a null that was never filtered, and a default placeholder used for unknowns. Remedies work by breaking the key up: salting it, broadcasting the small side of a join so no shuffle is needed, or filtering the pathological value out and handling it separately.

Shuffle

Operations that need all rows with a given key together — joins, group-by, distinct, sorts, repartitioning — cannot be done where the data happens to sit, so the engine writes intermediate data out, sends it across the network, and reads it back partitioned by key. That is a shuffle, and it costs disk, network and time in a way that per-row computation does not. The practical consequences are the heart of tuning a slow job: the fix is almost never a bigger machine, it is removing or shrinking the shuffle — filtering and projecting before the join rather than after, broadcasting a small table so the large one stays put, pre-partitioning data on its join key, and avoiding operations that reshuffle data you already had in the right place.

Data contract

Without a contract, every downstream consumer depends on whatever the producer happened to emit last week, so an innocuous source change breaks six reports and nobody knew they existed. A contract makes the dependency explicit and checkable: the columns and types, what each one means, when the data will be there, and the quality rules it satisfies — plus how a change will be communicated and how long the old shape stays supported. On AWS the enforcement is assembled from pieces rather than bought whole: the catalog carries the declared schema, a quality ruleset asserts the invariants on every run, and the orchestrator refuses to publish a run that fails them. The point is not documentation; it is that a violation stops the pipeline instead of reaching a dashboard.

Bounded and unbounded data

This is the distinction underneath the whole streaming-or-batch argument, and it is a property of the data, not of the tool. A bounded dataset — yesterday’s orders, a vendor’s nightly export — has a last record, so a job can read all of it, compute an exact answer and exit. An unbounded dataset — a clickstream, device telemetry, a database’s change log — has no last record, so any computation over it is a snapshot of a thing still arriving, and you must decide which records count: hence windows, event time and watermarks.

The practical test for a scenario question: can you name the moment the input is complete? If yes, batch is available and usually cheaper. If no, you are either streaming or you are inventing an artificial boundary (a nightly cutoff) and accepting the late records it strands.

Backpressure

A pipeline is a chain of stages with finite capacity. When arrival rate exceeds service rate at any stage, something has to give, and there are only three options: push back on the producer (reject, throttle, block), buffer the excess, or drop it. Reading a system’s backpressure behaviour tells you how it fails.

Amazon Kinesis Data Streams pushes back: writes past a shard’s 1 MB/s or 1,000 records/s get a ProvisionedThroughputExceededException, and it is the producer’s job to retry. Amazon SQS buffers: the queue grows, ApproximateAgeOfOldestMessage climbs, and nothing is lost until the retention period expires. Apache Flink propagates backpressure up its own operator chain until the source stops reading. A fire-and-forget HTTP producer with no retry drops.

The mistake to avoid in a design answer is treating a buffer as a fix. A buffer converts a throughput problem into a latency problem and buys time; it only becomes a fix if the average arrival rate is below capacity and the burst is finite. If the average exceeds capacity, every buffer in the world just postpones the same failure and makes it bigger.

Exponential backoff with jitter

Backoff alone is not enough, and the reason is the exam-relevant half. If a thousand throttled clients all sleep exactly 100 ms, 200 ms, 400 ms, they retry in synchronised waves — the service sees the same spike it just rejected, a pattern called a retry storm or thundering herd. Adding jitter (a random component to each delay) spreads those retries out, and that smoothing is what actually lets the backlog drain.

The AWS SDKs implement this for you. The default retry mode is standard; adaptive adds a client-side rate limiter that learns the throttling it is receiving and slows the client before it is rejected. Both are configurable with max_attempts and retry_mode in the shared config file, per-client in code, or via the AWS_RETRY_MODE environment variable. When a question describes a pipeline that collapses under throttling, the answer is rarely "raise the quota" first — it is usually that the client retries immediately, or retries without jitter, or retries a non-retryable error forever.

Idempotent write

Four shapes of it appear in data pipelines, and the exam rewards recognising which one a scenario admits. Deterministic keys: write to s3://bucket/dt=2026-10-21/part-0001.parquet and a re-run overwrites the same object rather than adding a second one — which is why overwrite-by-partition beats append for a job that may be retried. Conditional writes: a DynamoDB PutItem with a condition expression on the item’s own version, or an S3 conditional write, so the second attempt fails harmlessly instead of duplicating. Upsert on a natural key: MERGE into a table keyed on the record’s business id. Transactional commit: the sink makes the data and the metadata visible in one step, so a half-finished retry is invisible.

The anti-pattern to recognise instantly is append plus auto-generated surrogate key: every retry adds rows, nothing detects it, and the error surfaces weeks later as a count that is 0.3% too high.

Shard

A shard is two things at once, and conflating them is the commonest Kinesis error. It is a capacity unit — each shard accepts up to 1 MB per second or 1,000 records per second of writes, whichever it hits first, and serves 2 MB per second of reads shared across all classic consumers. It is also an ordering domain: records are sequenced within a shard, and nowhere else.

The second role is why you cannot think about shard count purely as arithmetic. Each shard owns a contiguous range of the 128-bit MD5 hash space, and a record lands in the shard whose range contains the hash of its partition key. So capacity is only usable if the keys spread across the ranges; a stream with ten shards and one busy key has the throughput of one shard. See hot shard.

Two numbers worth holding: GetRecords is limited to five calls per second per shard, so a single shard read by many classic consumers runs out of calls before it runs out of bytes — that is the case enhanced fan-out exists for. And a single record cannot exceed 1 MB.

Amazon Kinesis Data Streams developer guide — streams quotas and limits

Partition key

The producer supplies a partition key with every PutRecord. Kinesis hashes it with MD5 into a 128-bit value and routes the record to the shard whose hash-key range contains it. Two consequences the exam tests relentlessly:

What ordering you actually get. All records with the same partition key go to the same shard (until that shard is split) and are therefore read in the order they were written. Records with different keys give you no cross-key guarantee at all, even inside one shard. So "order events for one customer" is satisfied by customerId as the key; "order every event in the stream" is satisfied only by a single shard, which caps you at 1 MB/s.

Cardinality decides whether your capacity is reachable. A key with few distinct values, or a skewed distribution, concentrates traffic — the classic cases being deviceType, countryCode, a constant, or a tenant id in a system with one enormous tenant. The fix is a key with high cardinality and even spread, or a composite key such as tenantId#bucket where the bucket is a small random salt, which trades some ordering scope for a flat load.

A producer may instead supply an explicit hash key to place a record in a chosen shard directly. That is how the Kinesis Producer Library’s aggregation keeps aggregated records together.

Amazon Kinesis Data Streams developer guide — partition keys

Hot shard

The symptom is a stream that reports plenty of spare capacity in aggregate while producers receive ProvisionedThroughputExceededException and WriteProvisionedThroughputExceeded is non-zero. The cause is that a shard’s limits are per shard, and one key — or a small set of keys — is hashing into a single range.

Why resharding is the wrong first move: a split divides a hash range in two, and if every hot record carries the same key, that key’s hash falls in exactly one half. You now have two shards where one is still saturated and the other is empty, and you are paying for both. The repair is on the producer side: increase the key’s cardinality. Salt it (key#random(0..N)), make it composite, or pick a different field — and accept that you have narrowed the ordering guarantee to the salted key, which is the real cost of the fix and the thing a good exam answer acknowledges.

Diagnose it with enhanced shard-level monitoring, which publishes IncomingBytes and IncomingRecords per shard; without it you only see stream-level totals, which is precisely the view in which a hot shard is invisible.

Amazon Kinesis Data Streams developer guide — monitoring with shard-level metrics

Resharding

There are two primitives: SplitShard divides one shard’s hash range into two child shards, and MergeShards combines two adjacent ranges into one. UpdateShardCount is the convenience API that performs a series of splits or merges to reach a target count — provisioned mode only, and it recommends a target that is a multiple of 25% of the current count, because any other target takes longer to reach.

The mechanics that matter for a scenario question. A reshard does not move existing records: the parent shard is closed, keeps its data for the remainder of the retention period, and consumers must drain it before reading the children — which is why the Kinesis Client Library enforces parent-before-child ordering and why a hand-rolled consumer that ignores closed shards silently skips data. UpdateShardCount also creates short-lived intermediate shards that count against your account’s shard quota while they exist.

Default limits: no more than ten scaling operations per rolling 24 hours per stream, no scaling up beyond double or down below half the current count in one call, and a ceiling of 10,000 shards per stream without a quota increase. That daily cap is the reason reactive auto-scaling built on UpdateShardCount is a trap for a spiky workload — on-demand mode is the service’s own answer.

Amazon Kinesis Data Streams API reference — UpdateShardCount, SplitShard, MergeShards

On-demand capacity mode

On-demand streams have no shard count to choose. The service tracks your throughput and scales capacity automatically, doubling to accommodate a peak within a rolling period; you pay per stream-hour plus per GB written and read, with no shard-hour line. Provisioned mode keeps you in charge and is cheaper when load is high and steady, because you are buying capacity you will actually use.

The decision is about predictability, not volume. A new workload nobody has measured, a seasonal or bursty one, and anything where being throttled is worse than being slightly overcharged, all point to on-demand — in particular because reactive scaling in provisioned mode is capped at ten changes per rolling 24 hours, which cannot track a genuinely spiky day. A steady, well-understood, large stream points to provisioned.

Two caveats worth knowing: on-demand still partitions by partition key, so it does not rescue you from a hot shard — the per-shard limits still apply underneath; and UpdateShardCount is rejected on an on-demand stream, which is a tidy little trap for an automation script written against the wrong mode.

Amazon Kinesis Data Streams developer guide — choosing the data stream capacity mode

Enhanced fan-out

By default every consumer of a shard competes for the same 2 MB per second and the same five GetRecords calls per second. Add a third and a fourth application and they slow each other down; propagation delay climbs toward a second or more as each polls less often. Enhanced fan-out changes the model: you register a consumer, and Kinesis pushes records to it over a persistent HTTP/2 connection with its own dedicated 2 MB/s per shard, bringing typical propagation delay to around 70 ms.

When to reach for it: more than two or three consumers on the same stream, or a latency target the shared poll cannot meet. What it costs: per-consumer-shard-hour and per-GB retrieval charges on top of the stream, so it is the answer to a fan-out or latency requirement and not a default. What it does not do: nothing about write throughput, and nothing about a hot shard.

Amazon Kinesis Data Streams developer guide — developing enhanced fan-out consumers

Iterator age

Published as GetRecords.IteratorAgeMilliseconds by Kinesis and as IteratorAge by a Lambda event source mapping, it is the difference between now and the arrival time of the record just read. Zero means caught up. Rising means the consumer is processing slower than the producer is writing, and the number is the lead time you have left: when iterator age reaches the retention period, the oldest unread records are being deleted before anyone reads them, and that data is gone.

This is the alarm to build on a streaming pipeline, in preference to error rates — a consumer that is merely slow throws no errors at all. Four causes, in rough order of frequency: the consumer is slow per record (an external call in the hot path), the batch size is too small to amortise invocation overhead, a hot shard is funnelling the load into one worker, or a poison record is being retried forever and blocking its shard. The last one is why BisectBatchOnFunctionError, MaximumRetryAttempts and an on-failure destination exist on a Lambda event source mapping: without them one bad record stalls a shard until the data ages out.

Amazon Kinesis Data Streams developer guide — monitoring with Amazon CloudWatch

Stream retention period

Retention is the width of the replay window and the deadline for every consumer. The default is 24 hours. IncreaseStreamRetentionPeriod raises it to as much as 7 days at an extended-retention rate, and beyond that long-term retention keeps data for up to 365 days, billed per GB-month with its own per-GB charge for reads older than 7 days.

Three reasons the number is an architecture decision rather than a knob. It bounds replay: a bug found on Tuesday can only be fixed by reprocessing if Monday’s records still exist. It bounds recovery: a consumer down for longer than retention has a permanent hole, which is what makes iterator age the alarm that matters. And it bounds onboarding: a new consumer starting at TRIM_HORIZON can only see as far back as retention allows.

Note the contrast that the exam likes: Firehose has no retention and no replay — it buffers for up to 900 seconds and delivers. If a requirement says "two independent consumers, and we must be able to reprocess last week", Firehose alone cannot satisfy it.

Amazon Kinesis Data Streams developer guide — changing the data retention period

ProvisionedThroughputExceededException

Returned when a PutRecord or PutRecords would push a shard past 1 MB/s or 1,000 records/s. With PutRecords the failure is partial: the call returns HTTP 200 with a FailedRecordCount and a per-record error, so a producer that only checks the HTTP status loses data silently. That bug is common enough to be worth recognising on sight — the correct pattern is to re-submit only the failed records, with backoff and jitter.

Before adding shards, ask which shard. If WriteProvisionedThroughputExceeded is concentrated on one shard under enhanced shard-level monitoring, you have a hot shard and the fix is the partition key. If it is spread evenly, the stream genuinely needs capacity: reshard, or move to on-demand. If it appears only in short spikes, the Kinesis Producer Library’s aggregation can cut the record count by an order of magnitude and often removes the problem entirely, because the 1,000-records/s limit bites long before the 1 MB/s one on small records.

Amazon Kinesis Data Streams developer guide — handling PutRecords failures

KPL record aggregation

Two distinct batchings happen in the Kinesis Producer Library, and the exam occasionally tests the difference. Collection is the ordinary one: many Kinesis records sent in a single PutRecords HTTP call, which saves round trips. Aggregation packs many user records into the payload of a single Kinesis record, up to the 1 MB record ceiling — so a shard that could take 1,000 records per second can take tens of thousands of user records per second, because Kinesis counts the container, not the contents.

The cost is that consumers must deaggregate. The Kinesis Client Library does it transparently, and so does a Lambda event source mapping; a hand-written GetRecords consumer does not, and will read one opaque protobuf blob where it expected a record. Aggregation also adds producer-side latency equal to RecordMaxBufferedTime, because the library has to wait to fill a container, which is the trade a latency-sensitive stem is usually pointing at.

Amazon Kinesis Data Streams developer guide — KPL key concepts: aggregation and collection

Replay

Replay is the capability that distinguishes a log from a pipe. Kinesis Data Streams and Apache Kafka keep records for a configured retention period and let a consumer start at a sequence number, a timestamp, the oldest available record (TRIM_HORIZON / earliest) or the newest (LATEST). Firehose and SQS do not: Firehose has delivered and forgotten; SQS deletes on acknowledgement.

Why it is a design requirement and not a nice-to-have. A transformation bug means every downstream record derived from the stream is wrong, and the only honest repair is to recompute from the source — which needs the source to still exist. This is also the architectural argument for landing raw records in Amazon S3 as the first step of a pipeline: S3 retention is effectively unlimited, so the raw zone becomes the replay source once the stream’s window has closed. A pipeline whose only copy of the truth is a 24-hour stream has a 24-hour statute of limitations on its own bugs.

Parallelization factor

Set on the event source mapping for Kinesis and DynamoDB Streams. By default one Lambda invocation processes one shard at a time, so throughput per shard is capped by your function’s duration. Raising the factor lets Lambda poll several batches from the same shard at once: a factor of 10 over 5 shards gives up to 50 concurrent invocations.

The detail that makes it safe, and that the exam checks: Lambda groups the batches by partition key, so all records sharing a key still go to the same concurrent processor in order. You lose no per-key ordering by raising the factor — you only lose ordering across keys within a shard, which you never had a guarantee on anyway.

It is the cheap first move when iterator age is climbing and the consumer is CPU- or IO-bound, because it needs no resharding and no producer change. It does not help if the backlog is caused by one dominant key (all its records are in one batch group) and it does not help if the function is failing, where the answer is error handling instead.

AWS Lambda developer guide — using Lambda with Kinesis Data Streams

Amazon MSK

MSK runs open-source Apache Kafka, which is the whole point of choosing it: existing producers, consumers, Kafka Streams applications and connectors work without modification, and so does the operational vocabulary — topics, partitions, consumer groups, offsets, replication factor, min.insync.replicas. AWS owns provisioning, broker patching, Apache ZooKeeper or KRaft metadata, and the CloudWatch integration; you still own partition counts, topic configuration and client tuning.

The honest comparison with Kinesis Data Streams is not about capability — both are durable, ordered, replayable logs — it is about what you are optimising for. Pick MSK for Kafka API compatibility, an existing Kafka estate, the broader connector ecosystem, or per-topic configuration that Kinesis does not expose. Pick Kinesis for a smaller operational surface, native Lambda and Firehose integration, and capacity you never size. A stem that mentions migrating an on-premises Kafka cluster, or "our team already runs Kafka Connect", is pointing at MSK; one that says "minimise operational overhead" with no existing Kafka is usually pointing at Kinesis.

Authentication options differ too, and get tested: IAM access control, SASL/SCRAM backed by AWS Secrets Manager, and mutual TLS with AWS Private CA.

Amazon Managed Streaming for Apache Kafka developer guide

Kafka partition

A topic is split into partitions; a record’s key is hashed to choose one; records within a partition are strictly ordered and addressed by a monotonically increasing offset. So far this is the shard story with different nouns, and the same skew rules apply.

Three differences matter. Partition count only goes up. kafka-topics --alter --partitions can increase it; nothing decreases it, so an over-partitioned topic is a permanent cost in open file handles, replication traffic and metadata. Adding partitions breaks key affinity. The hash is taken modulo the partition count, so increasing the count re-maps existing keys to different partitions and the ordering guarantee for a key spans the change discontinuously — records for one key can be read out of order across the boundary unless you drain first. Durability is yours to configure, via replication factor and min.insync.replicas together with producer acks; Kinesis replicates across three Availability Zones with no setting to get wrong.

Consumption parallelism is capped by partition count: within one consumer group, a partition is assigned to exactly one consumer, so a topic with 6 partitions cannot usefully be read by 10 consumers — four sit idle.

Apache Kafka documentation — topics and partitions (as deployed by Amazon MSK)

Consumer group

Kafka’s fan-out model. Every consumer declares a group.id; the group’s members divide the topic’s partitions between them, and the group commits its own offsets. Two different groups reading the same topic therefore progress independently — application A can be at offset 900 while application B replays from 0 — which is Kafka’s equivalent of registering a second Kinesis consumer.

Two behaviours a troubleshooting question leans on. Rebalancing: when a member joins, leaves or stops heartbeating, partitions are reassigned, and during the rebalance the group stops consuming; a consumer whose processing time exceeds max.poll.interval.ms is ejected, triggering a rebalance, which slows everyone, which ejects the next one — a feedback loop that reads as "our consumers keep restarting". Consumer lag is the offset gap between the log end and the group’s committed offset, the Kafka analogue of iterator age, and the thing to alarm on.

MSK Connect

Kafka Connect is the framework for moving data between Kafka and everything else with configuration rather than code — a source connector pulls from a database or queue into a topic, a sink connector pushes a topic into S3, OpenSearch, a warehouse. MSK Connect runs those same community or custom connector plugins as a managed, scalable connector resource, so you upload the plugin and a configuration and AWS runs the workers.

Its role in a design answer is the "we already have a connector for that" path: a stem describing an existing Debezium change-capture connector, or an S3 sink the team has used for years, is usually better served by MSK Connect than by rewriting the integration as a stream-processing application. What it is not is a transformation engine — single-message transforms handle field renames and routing, not joins or windowed aggregation, which is where Managed Service for Apache Flink starts.

Amazon MSK developer guide — MSK Connect

MSK Serverless

Provisioned MSK asks you to choose broker instance types, broker counts and storage per Availability Zone — the normal Kafka capacity exercise. Serverless removes it: you create a cluster, create topics, and the service handles capacity, with per-partition throughput managed for you.

The trade-offs are the ones you would expect, and they are what a question turns on. Serverless supports IAM authentication only, where provisioned also offers SASL/SCRAM and mutual TLS, so a stem whose clients authenticate with SCRAM rules it out. Serverless has lower per-partition throughput ceilings and tighter quotas on partitions per cluster, so a very large steady workload belongs on provisioned. And provisioned clusters support tiered storage for long retention; serverless offers unlimited retention on its own terms. The usual reading: unpredictable or spiky traffic and a team that does not want to size brokers → serverless; large, steady, SCRAM or mTLS, or custom broker configuration → provisioned.

Amazon MSK developer guide — MSK Serverless

MSK tiered storage

Without it, Kafka retention is bounded by the EBS volumes attached to your brokers, so "keep a year of events" means provisioning — and paying for, and rebalancing — a year of broker storage. Tiered storage splits the log: a configurable local retention window lives on the broker for low-latency reads, and everything older is moved to a remote tier at a much lower price per GB with no effect on the client API. A consumer reading from the beginning of a topic transparently reads remote segments.

In an exam answer it is the cost lever for long retention on MSK, the counterpart of Kinesis long-term retention. The caveats: it is available on provisioned clusters (not serverless), reads from the remote tier are slower than from a broker disk, and you set two retention values — the local one and the overall one — which is the configuration people get backwards.

Amazon MSK developer guide — tiered storage

Firehose buffering hints

Firehose accumulates records and flushes a batch when either threshold is reached — so the buffer interval is a latency ceiling and the buffer size is a file-size floor, and together they are the knob that decides whether you get the small-files problem downstream. A 60-second interval on a quiet stream produces 1,440 tiny objects per day per partition; a 128 MB / 900-second setting produces a handful of well-sized ones at the cost of up to fifteen minutes of delay.

Three specifics worth memorising, because they are exactly the sort of thing a stem is built on. Enabling Parquet or ORC record format conversion or dynamic partitioning raises the minimum buffer size to 64 MB, with 128 MB the default. Zero buffering is unavailable with dynamic partitioning. And with dynamic partitioning Firehose buffers in several internal stages, so end-to-end delay can be about 1.5× the configured interval — a pipeline promised "60-second freshness" can legitimately deliver in 90.

Firehose also raises the buffer size on its own when delivery falls behind ingestion, to catch up. That is the service protecting durability, and it means the file sizes you observe in production can exceed the ones you configured.

Amazon Data Firehose developer guide — buffering hints

Dynamic partitioning

Without it, a Firehose stream writes to a prefix based on arrival time (YYYY/MM/DD/HH), which is the ingestion date — fine for a log archive, wrong for anything partitioned by a business attribute. Dynamic partitioning extracts keys from the record with a JQ expression or from a Lambda transform, and substitutes them into the prefix: customer_id=!{partitionKeyFromQuery:customer_id}/dt=!{...}/.

Four limits that decide whether it fits. Each active partition holds its own buffer, and there is a ceiling of 500 active partitions per Firehose stream — exceed it and the overflow records are written to the configured S3 error prefix with activePartitionExceeded, which is data in the wrong place rather than data lost. The minimum buffer size becomes 64 MB. Zero buffering is not supported. And JQ-based partitioning requires JSON input, so a binary or multi-record payload needs the Lambda route.

The practical reading: dynamic partitioning is for a handful of low-cardinality attributes — date, region, event type, tenant in a small estate. A high-cardinality key such as user id will exhaust the active-partition ceiling, and the right answer there is to land the data flat and partition it in a downstream compaction job.

Amazon Data Firehose developer guide — dynamic partitioning

Record format conversion

Turned on per Firehose stream. The service reads the target table definition from the Glue Data Catalog and serialises each buffered batch as Parquet or ORC instead of raw JSON, which typically cuts stored bytes several-fold and lets Athena skip columns it does not reference.

The constraints are the exam material. Input must be JSON — anything else needs a Lambda transform first to produce JSON. A Catalog table is mandatory, so there is a hard ordering dependency: no table, no conversion. Minimum buffer size rises to 64 MB. And conversion is applied per batch, so the file-size benefit still depends on buffering hints: converting a 1 MB buffer to Parquet gives you a small Parquet file, which has worse read characteristics than one large JSON file because each file carries footer metadata and a minimum read overhead.

The decision it replaces is a downstream Glue job that reads JSON and writes Parquet. Conversion in Firehose removes that job entirely for append-only data; keep the job when you also need to join, deduplicate or repartition, because Firehose cannot do any of those.

Amazon Data Firehose developer guide — converting input record format

Tumbling window

Each record belongs to exactly one window, windows are contiguous and equal in length, and an aggregate is emitted when a window closes. "Orders per minute" is a tumbling window. Contrast a sliding (hopping) window, which advances by less than its length so records belong to several windows — "the five-minute average, updated every minute" — and a session window, which groups records separated by less than a gap of inactivity and so has no fixed length at all.

Where they appear on this exam: Managed Service for Apache Flink and Spark Structured Streaming implement all three; a Lambda event source mapping on Kinesis supports a tumbling window of up to 15 minutes, passing aggregate state between invocations, which is the cheap way to do per-key counting without a Flink application; and Firehose has no windowing at all, only buffering, which is a different thing — a buffer flushes on size or time and makes no promise about which records are in it.

The subtlety that separates a correct answer from a plausible one is which clock the window uses. See event time.

Watermark

An event-time window over an unbounded stream faces an impossible question: when may it emit a result, given that a late record could always turn up? A watermark answers it by fiat. The engine tracks the maximum event timestamp it has seen, subtracts an allowed lateness, and declares that windows ending before that point are complete and may fire.

So the watermark is a tuning knob on a correctness-versus-latency trade, not a fact. A generous allowed lateness means accurate windows that report late and operator state held in memory for longer. A tight one means fast results and dropped stragglers. Flink lets you route late records to a side output rather than discarding them, which is how a pipeline can be both prompt and eventually correct: emit on the watermark, then reconcile the side output into the stored result.

A stem that says "a small percentage of events arrive several hours late and must still be counted" is not describing a windowing problem at all — hours of allowed lateness is not operable. It is describing a two-path design: stream for the prompt approximate answer, and a periodic batch recomputation over the raw landing zone for the authoritative one.

Checkpointing and savepoints

A stateful stream processor holds running aggregates in memory. If it crashes, that state is gone — and simply re-reading from the last committed offset would recount records already folded into the lost state. A checkpoint is a consistent snapshot of all operator state and the source positions that produced it, taken automatically on an interval; on recovery Flink restores the state and rewinds the sources to the matching offsets, which is what makes exactly-once state updates possible.

A savepoint is the same mechanism invoked deliberately and retained: you take one before stopping an application, change the code or the parallelism, and resume from it. In Amazon Managed Service for Apache Flink these are surfaced as snapshots, taken automatically when enabled and on your own command.

Two operational facts. Checkpoints cost throughput while they are taken, so the interval is a tuning decision, and very large keyed state makes them slow. And an application whose business logic changed incompatibly cannot resume from an old savepoint — the state schema no longer matches — which is why a deployment plan for a stateful stream job includes "can we restore, and from what".

Amazon Managed Service for Apache Flink developer guide — fault tolerance and snapshots

Change data capture (CDC)

The alternative to CDC is a periodic query — SELECT * WHERE updated_at > :last_run — and it fails in four specific ways that CDC does not. It misses deletes entirely, because a deleted row matches no query. It misses any update that does not touch the watermark column. It loads the source: a full or large incremental scan competes with production traffic exactly when the table is biggest. And it has a race at the boundary — a transaction committing during the scan can be read half-done or skipped.

CDC reads the log the database already writes for its own durability, so it sees every committed change in commit order, including deletes, with far less load on the source. Its costs are different: the log must be retained long enough for the reader to keep up, the source needs configuration (supplemental logging on Oracle, row-based binary logging on MySQL, a logical replication slot on PostgreSQL), and the output is a stream of change events whose application is now your problem — see applying changes downstream.

On AWS this is AWS DMS for relational sources, DynamoDB Streams for DynamoDB, a Debezium connector on MSK Connect for a Kafka-native estate, and zero-ETL integrations where the pair is supported.

AWS DMS

A DMS task has a source endpoint, a target endpoint and a replication instance (or serverless capacity) that does the work. Three task types: full load copies the existing rows once; CDC only starts from a point in the log and replicates changes; full load plus CDC does both, capturing changes during the initial copy and applying them afterwards so the target converges without a maintenance window.

Its range is wider than its name suggests and the exam exploits that. Sources include the commercial and open-source relational engines plus MongoDB, DocumentDB and S3; targets include Amazon S3, Kinesis Data Streams, Amazon MSK, OpenSearch and Redshift as well as databases. "DMS into S3 as Parquet, partitioned by date" is a perfectly ordinary ingestion answer, not a migration one.

One limitation to keep: views can be migrated only by a full-load task. A CDC or full-load-plus-CDC task carries tables only, because a view has no rows of its own in the change log.

AWS Database Migration Service user guide — creating tasks for ongoing replication

DMS ongoing replication

DMS uses each engine’s own mechanism — Oracle LogMiner or the binary reader over redo logs by system change number, SQL Server’s transaction log via MS-Replication or MS-CDC read with fn_dblog, MySQL row-based binary logs, PostgreSQL logical replication slots with the test_decoding plugin. AWS states plainly that CDC does not provide real-time replication and carries no latency SLA; latency can reach minutes or longer depending on source workload, network, instance size and target ingest rate.

The mechanism behind the most-tested failure. DMS forwards only committed transactions, to keep the target consistent. So a long-running open transaction on the source blocks the task: DMS cannot apply anything after the transaction’s start until it commits, CDCLatencySource climbs from seconds to hours, CDCIncomingChanges accumulates and changes buffer to memory then disk. A stem describing replication lag that spikes every night at the same time, with no change in volume, is describing a nightly batch job that holds a transaction open — and the fix is on the source, not in a bigger replication instance.

The two latency metrics to distinguish: CDCLatencySource is the gap between the source and DMS (reading problem), CDCLatencyTarget is the gap between DMS and the target (applying problem). They point at opposite remedies.

AWS Database Migration Service user guide — ongoing replication, and CDC latency metrics

Supplemental logging

Oracle’s redo log exists to recover that database, so by default it records the minimum needed to redo a change locally — often just a row identifier and the changed values. A replication target needs more: the key columns that identify the row in its own copy, and in some configurations every column. Supplemental logging is the instruction to include it, set at database level and optionally per table.

The reason it belongs in a data engineer’s vocabulary is that it is a prerequisite on the source that an AWS-side configuration cannot substitute for. A DMS task against Oracle will do a full load and then fail or silently drop updates on CDC if supplemental logging is missing — and the symptom, "the initial copy worked and then nothing changed", is a favourite stem. MySQL has the direct equivalent: binary logging must be enabled and set to ROW format, because statement-based logging records the SQL rather than the resulting rows and cannot be replayed faithfully against a different engine. PostgreSQL needs wal_level = logical and replication slots, which have the opposite hazard: an inactive slot retains write-ahead log segments indefinitely and can fill the source’s disk.

AWS Database Migration Service user guide — using an Oracle database as a source

Zero-ETL integration

An integration you create once: AWS keeps the target continuously updated from the source’s changes, with seconds-to-minutes freshness, and there is no job, no connector, no replication instance and no code to own. The supported pairs have grown steadily; the ones to recognise are Aurora MySQL and PostgreSQL to Amazon Redshift, RDS for MySQL to Redshift, and DynamoDB to Redshift and to OpenSearch Service.

Why it is the right answer when it is: the requirement is "analytics on operational data, kept fresh, with minimal operational overhead", and the pair happens to be supported. Why it often is not: zero-ETL replicates — it does not transform. There is no place to join a second source, apply business logic, deduplicate, mask a column or reshape a schema. A stem that needs any of those needs a pipeline, and offering zero-ETL is then offering the wrong shape of thing however fashionable it is. It is also pair-specific: no integration exists for an on-premises Oracle database, which is where DMS remains the answer.

Amazon Redshift management guide — working with zero-ETL integrations

DynamoDB Streams

Enabled per table, with a view type that decides what each record carries: KEYS_ONLY, NEW_IMAGE, OLD_IMAGE or NEW_AND_OLD_IMAGES. The last is the one that makes genuine change processing possible, because only by comparing the images can a consumer tell which attributes changed. Records are ordered per partition key and retained for 24 hours — a fixed window, unlike a Kinesis stream’s configurable one.

The standard consumer is a Lambda event source mapping, which brings parallelization factor, batching windows, iterator age and error-handling controls along with it. The standard pipeline is Streams → Lambda → somewhere analytical, or Streams → Firehose → S3 via EventBridge Pipes.

Distinguish it from Kinesis Data Streams for DynamoDB, a separate feature that sends the same change records to a Kinesis stream you own — which buys you longer retention, multiple independent consumers and enhanced fan-out, at the cost of running a stream. When a stem says "retain changes for more than 24 hours" or "several teams need the change feed", that is the one it means.

Amazon DynamoDB developer guide — change data capture with Streams

Applying change events downstream

This is the step people forget when they design a CDC pipeline, and it is where the pipeline actually gets hard. The feed is a sequence of inserts, updates and deletes; the warehouse or lake table has to reflect the current state, or a history of states. Three patterns, each with a cost:

Append and read the latest. Write every change event as a new row with its commit sequence, and define a view that picks the newest row per key. Writes are trivial and idempotent; reads pay a window function over every version forever, and the table grows without bound.

Merge on a key. MERGE or an equivalent upsert into the target, deleting on a delete event. Reads are cheap and the table stays small; writes need a transactional target — an open table format such as Apache Iceberg, or a warehouse — because a merge over plain S3 files means rewriting them.

Keep the history deliberately. Close the previous row with an end timestamp and insert a new one, so the table can answer "what did this look like in March". This is a type-2 dimension, and the cost is that every query must filter on the validity window or it will count a customer several times.

The decision is driven by whether anyone needs history and whether the target can transact. Order matters in all three: change events must be applied in commit order per key, which is exactly the ordering guarantee a partition key on the primary key buys you.

AWS Transfer Family

A great deal of real data arrives because a bank, insurer or logistics partner drops a file on an SFTP server every night, and will not be persuaded otherwise. Transfer Family gives you that endpoint without a server: the protocol terminates in AWS, the files land in S3 or EFS, and authentication can be service-managed keys, a custom identity provider, or AWS Directory Service.

The design point is that it is an ingestion door, not a pipeline. A file arriving is an S3 event, which is the trigger for whatever processes it. The other half worth knowing is managed workflows: a Transfer Family workflow can run steps on upload — copy, tag, decrypt a PGP file, call a Lambda — which handles the common "the partner sends it encrypted and gzipped" case without a separate orchestration.

Distinguish it from the neighbours a question will offer alongside: DataSync moves large volumes between file systems and AWS on a schedule you control, Storage Gateway presents AWS storage to on-premises applications as if it were local, and Transfer Family exists specifically to speak a file-transfer protocol to a third party.

AWS Transfer Family user guide

AWS Snow Family

The decision is arithmetic and you should be able to do it in your head: a 100 Mbps link moves roughly 1 TB per day at perfect utilisation. So 500 TB over a dedicated 1 Gbps link is about 50 days of continuous saturated transfer, during which that link does nothing else — and that is the number that makes a courier competitive. AWS Snowball Edge devices come in storage-optimised and compute-optimised variants, and a job can use several in parallel.

Two details a stem can hinge on. The data is encrypted with AWS KMS keys and the device is tamper-evident, so "the data cannot traverse the public internet" is satisfied. And compute-optimised devices can run EC2 instances and Lambda functions locally, which is the answer to "pre-process or filter at a site with no usable connectivity" rather than to a pure transfer question.

The comparison the exam wants: continuous or incremental transfer over a network → DataSync; a one-time bulk move whose network time is unacceptable → Snow; a partner pushing files on a protocol → Transfer Family; and a dedicated private circuit for sustained high volume → AWS Direct Connect, which changes the arithmetic rather than avoiding it.

AWS Snowball Edge developer guide

Multipart upload

A single PutObject is capped at 5 GB and is all-or-nothing: a dropped connection at 4.9 GB means starting again. Multipart upload initiates an upload id, sends parts of 5 MB or more (the last may be smaller, maximum 10,000 parts, maximum 5 TB object), and completes with a list of part numbers and ETags. Parts travel in parallel, a failed part is retried alone, and an upload can be paused and resumed.

Two things to carry into a cost or troubleshooting question. Incomplete multipart uploads are billed — the parts sit in the bucket, invisible in a normal object listing, accruing storage charges until they are aborted. The fix is an S3 Lifecycle rule with AbortIncompleteMultipartUpload, and a bucket whose storage cost does not match its visible contents is very often this. And the AWS CLI and SDK transfer managers use multipart automatically above a threshold you can tune, which is why aws s3 cp of a huge file behaves well without being told to.

Amazon S3 user guide — uploading and copying objects using multipart upload

Amazon AppFlow

The problem AppFlow addresses is unglamorous and extremely common: the data you need is behind a vendor’s REST API with its own pagination, OAuth dance, rate limits and incremental-change semantics, and writing that client is a month of work you will then maintain forever. AppFlow ships the connector. You configure a flow with a source object, a destination (S3, Redshift, Snowflake, Salesforce and others), a trigger (on demand, scheduled, or on a source event), optional field mapping, filtering and masking.

In an exam it is the right answer to a stem that names a SaaS application as the source and emphasises minimal custom code. It is the wrong answer when the source is a database (that is DMS or a Glue connection), when the volume is very large, or when the transformation needed is more than field-level — AppFlow maps and filters, it does not join.

Amazon AppFlow user guide

Glue connection

A Glue job that reads a database needs three things it cannot invent: the endpoint and driver, the credentials, and a network path. A connection packages all three as a Catalog object, which is why the same connection can be shared by a crawler, several jobs and a Glue Studio visual editor.

The network half is the part that generates support tickets, and the mechanism is worth understanding. A Glue job attached to a connection runs its Spark environment with elastic network interfaces in the subnet named by that connection, so it has the connectivity of that subnet and nothing more. A private subnet with no route to Amazon S3 therefore produces a job that reaches the database and then hangs reading its own script or writing output — the fix is an S3 gateway VPC endpoint, which is the single most common Glue networking answer. The security group must also allow a self-referencing inbound rule, because Spark executors talk to each other.

A job needing two data stores in different VPCs cannot attach to both unless the networks are joined; the usual answer is to land one side in S3 first rather than to force the connectivity.

AWS Glue developer guide — connecting to data

Dead-letter queue

Without one, a poison record is retried forever. On an SQS queue that means the message returns to visibility again and again while the backlog behind it grows; on a Kinesis or DynamoDB Streams event source mapping it is worse, because the shard is ordered — the failing batch is retried in place and nothing after it is processed, so iterator age climbs until the data ages out.

The controls differ by source and the exam distinguishes them. An SQS queue takes a redrive policy with a maxReceiveCount. A stream event source mapping takes MaximumRetryAttempts, MaximumRecordAgeInSeconds, BisectBatchOnFunctionError — which halves a failing batch repeatedly to isolate the single bad record instead of discarding 10,000 good ones with it — and an OnFailure destination that receives metadata about the failed batch, not the records themselves, since the records are still in the stream.

The operational rule that follows: a dead-letter queue with no alarm on its depth is a silent data-loss mechanism. The queue is the place failures become visible, so visibility has to be wired up, and a non-empty one is a work item rather than a resolved problem.

Amazon SQS developer guide — dead-letter queues

Visibility timeout

SQS does not delete a message when it is read — it hides it. The consumer must call DeleteMessage after succeeding; if it does not, the timeout expires and the message becomes visible to someone else, which is exactly the behaviour you want when a worker dies mid-task. Default 30 seconds, maximum 12 hours, extendable in flight with ChangeMessageVisibility.

The classic misconfiguration, and a reliable stem: a Lambda consumer with a 5-minute timeout on a queue with a 30-second visibility timeout. At 30 seconds the message reappears, a second invocation starts the same work, and the two race — the symptom being duplicated downstream writes with no error anywhere in the logs. AWS’s own guidance is to set the queue’s visibility timeout to at least six times the function timeout when batching, and the general rule is that it must exceed the worst-case processing time plus the retry budget.

This is also the cleanest illustration of why idempotency is not optional: the message will occasionally be delivered twice, by design, because the alternative is losing it.

Amazon SQS developer guide — visibility timeout

SDK retry mode

Every AWS SDK retries some failures for you, and knowing which mode is in force explains behaviour that otherwise looks like magic or like a bug. standard is the current default: a fixed maximum of three attempts, a consistent set of retryable error codes across SDKs, and exponential backoff with jitter. adaptive adds a token-bucket rate limiter that measures the throttling the client is receiving and slows outgoing calls before they are rejected — useful for a bulk job hammering one API, and explicitly not recommended as a default because it will deliberately delay your calls. legacy is the older per-SDK behaviour.

Configure with max_attempts and retry_mode in the shared config file, the AWS_MAX_ATTEMPTS and AWS_RETRY_MODE environment variables, or per client in code. Two consequences for a data pipeline: a script that appears to hang on a throttled API is usually backing off, not stuck; and a PutRecords or BatchWriteItem call whose partial failures you do not re-submit is not covered by any retry mode at all, because the SDK saw a successful HTTP response.

AWS SDKs and Tools reference guide — retry behavior

Narrow transformation

Narrow transformations are the cheap ones, and the reason is structural rather than algorithmic: every row an output partition needs is already in the input partition sitting on that executor. Spark can therefore chain many narrow operations into a single stage and run them as one pass per partition, with no coordination between executors — filter, then project, then derive a column, all in one traversal of the data.

This is why "add more executors" works for narrow work and not for its opposite. Doubling the executors halves the wall-clock time of a filter, because the partitions are independent. See wide transformation for the case where it does not.

Wide transformation

To compute a total per customer, every row for that customer must end up on the same executor. Since the rows began scattered across every partition, Spark must physically move them: each task writes its rows into per-destination buckets on local disk, every reader fetches its bucket from every writer, and only then can the aggregate be computed. That exchange is the shuffle, and it is a stage boundary — nothing downstream starts until the shuffle for the stage is complete.

The cost is N writers × M readers network fetches plus a full write and re-read of the data through local disk, and it is why a job that reads 100 GB can move far more than 100 GB. The practical consequences are unintuitive enough to be worth stating flatly: adding executors makes a shuffle wider, not faster, because the number of connections grows with the number of writers and readers; and the only real remedies are to move less data (filter and project before the shuffle), to avoid the exchange entirely (a broadcast join), or to fix the distribution (skew).

Shuffle

The mechanism in order. Each task in the upstream stage partitions its output rows by the hash of the shuffle key into as many buckets as there will be reduce partitions — spark.sql.shuffle.partitions, default 200 — and writes those buckets as shuffle files on the executor’s local disk, with an index. The downstream stage’s tasks then each fetch their one bucket from every upstream task. So the data is serialised, written, transferred and deserialised, and the number of fetches is writers × readers.

Four symptoms this explains, all of them common exam stems. A job whose Spark UI shows a huge "Shuffle Read" against a modest input — the shuffle amplified it. No space left on device — shuffle files filled the executor’s disk, which is why Glue worker types advertise their disk size and why spill matters. A stage where 199 tasks finish in seconds and one runs for an hour — skew. And a job that gets slower when you give it more workers — more writers and readers, more connections, same data.

The shuffle partition count is a genuine tuning knob in both directions: too few and each partition is too big to fit in memory, too many and you pay scheduling overhead for thousands of tiny tasks. Adaptive query execution is Spark’s attempt to choose it for you at runtime — and note the trap that setting the property explicitly switches the automatic coalescing off.

Amazon EMR release guide — optimize Spark performance

Broadcast join

A sort-merge join shuffles both sides by the join key. If one side is small — a dimension table, a lookup, a date spine — it is far cheaper to send a complete copy to every executor and let each one join its local partition of the big side in memory. No exchange, no stage boundary, no skew, and the big table is never moved.

Spark will do this automatically when it can estimate the small side under spark.sql.autoBroadcastJoinThreshold, which defaults to 10,485,760 bytes (10 MiB), and adaptive join conversion widens that by re-checking the runtime size after filtering rather than relying on a plan-time estimate. You can force it with a broadcast() hint when you know the statistics are wrong, which is common over S3 data with no table statistics collected.

The failure mode to respect: broadcasting is bounded by executor memory and by the driver, which collects the small side before distributing it. Broadcasting something that is not actually small gives you an out-of-memory error on the driver or a Cannot broadcast the table that is larger than... failure — so "add a broadcast hint" is a fix with a size condition attached, not a universal one.

Amazon EMR release guide — adaptive join conversion

Adaptive query execution (AQE)

A cost-based optimiser has to guess sizes before the query runs, and over files in Amazon S3 with no collected statistics it guesses badly. AQE defers the decision: after each shuffle stage completes, Spark has exact sizes, and it re-optimises what remains. Three optimisations, all enabled by default in AWS Glue 3.0 and later and in current EMR releases:

Coalescing shuffle partitions. A stage configured for 200 partitions whose data turns out to be small gets its contiguous small partitions merged into fewer, larger tasks, removing scheduling overhead. The trap, and it is a real one: this is enabled by default unless spark.sql.shuffle.partitions is explicitly set — so hand-tuning the partition count switches off the mechanism that would have tuned it, and a job "optimised" that way can get slower.

Adaptive join conversion. A sort-merge join whose runtime side size comes in under spark.sql.autoBroadcastJoinThreshold is rewritten as a broadcast hash join, which is how a join over a heavily filtered table avoids a shuffle that the plan-time estimate said was necessary.

Skew join optimisation (spark.sql.adaptive.skewJoin.enabled). Outlier partitions are detected from shuffle file statistics, split into roughly even subpartitions, and the matching side is replicated to each — the automatic version of salting.

The master switch is spark.sql.adaptive.enabled; turning it off disables the framework and all three.

Amazon EMR release guide — adaptive query execution

Executor

A Spark application has one driver — which builds the plan, schedules tasks, and holds the results of anything you collect() or broadcast — and many executors, each with a fixed number of cores and a memory allocation. One task runs per core, and a task processes one partition.

That gives the arithmetic that answers most sizing questions: total parallelism is executors × cores per executor, and a job with 2,000 partitions on 40 cores runs 50 waves of tasks. In AWS Glue the units are packaged: a G.1X worker is 1 DPU (4 vCPU, 16 GB) running one executor with 4 cores, a G.2X is 2 DPU (8 vCPU, 32 GB) with one 8-core executor, and so on up the range — so "more workers" and "bigger workers" buy parallelism and per-task memory respectively, which are different cures.

The distinction that catches people is driver versus executor memory. Listing a million S3 objects, collecting results, or broadcasting a table stresses the driver; processing a huge partition stresses an executor. A driver out-of-memory error is not fixed by more workers — see the small-files problem, which is the usual cause.

AWS Big Data Blog — scale your AWS Glue for Apache Spark jobs with larger worker types

Spill

Spark does not fail when a sort or an aggregation exceeds the memory budget; it spills the excess to the executor’s local disk and merges it back. That is good engineering — the job completes — and it is also the single most common reason a job is slow rather than broken, because disk is far slower than memory and the data is written and read again.

How to recognise it: the Spark UI reports "Spill (Memory)" and "Spill (Disk)" per stage, and a stage with large disk spill is the one to fix. The causes are the familiar three: partitions too large (too few shuffle partitions), one partition disproportionate (skew), or a worker type whose memory is too small for the partition size. Taken far enough, spill becomes a hard failure — No space left on device when shuffle files and spill exhaust the volume, which is why the disk figure in the Glue worker table is a real constraint and not trivia.

The order to try remedies: more shuffle partitions (smaller pieces) before a bigger worker (more memory per piece), since the first costs nothing.

DPU (data processing unit)

A DPU is how Glue prices Spark capacity, and it is deliberately the same shape as a modest machine so that the arithmetic is easy: a job with 10 G.1X workers is 10 DPU, and running for 6 minutes costs 60 DPU-minutes. Billing is per second with a 1-minute minimum per run, which is the detail that makes thousands of tiny jobs expensive relative to the work they do.

The practical reading for a cost question, and it has two halves that are often collapsed into one wrong rule.

More parallelism is not inherently dearer. DPU-hours are DPU count x elapsed time, so for work that divides cleanly the product is unchanged: 10 workers for 10 minutes and 20 workers for 5 minutes are both 100 worker-minutes. You bought elapsed time, not capacity, and the bill is the same.

What is dearer is parallelism you cannot use. If elapsed time does not fall when you add workers — a shuffle-bound stage, or a skewed one where a single task is the stage — then the DPU-hours rise for nothing. That is the case the simulation in this module measures: doubling executors on a skewed key moved the wall clock by 1.08x, so the capacity-seconds nearly doubled for no gain. The test before adding workers is therefore whether the work actually divides, and halving the runtime by removing a shuffle is free money either way. And Flex execution is a discount on the same DPUs in exchange for a weaker start-time and interruption promise, which fits a nightly job and not an hourly SLA.

A Python shell job is sized in fractions — 0.0625 or 1 DPU — which is the right vehicle for a small orchestration or API-calling script that does not want a Spark cluster at all.

AWS Glue developer guide — AWS Glue pricing and DPU allocation

Glue worker type

Each worker type fixes vCPU, memory and local disk together, and each node runs exactly one Spark executor whose core count equals its vCPU count. The ladder, verified against the AWS documentation: G.025X = 0.25 DPU, 2 vCPU, 4 GB; G.1X = 1 DPU, 4 vCPU, 16 GB, 94 GB disk; G.2X = 2 DPU, 8 vCPU, 32 GB, 138 GB; G.4X = 4 DPU, 16 vCPU, 64 GB, 256 GB; G.8X = 8 DPU, 32 vCPU, 128 GB, 512 GB; and G.12X and G.16X above that on Glue 4.0 and 5.0. The R series trades CPU for memory at the same DPU count.

Choosing between "more workers" and "bigger workers" is the question this answers, and the two are not interchangeable. More workers buys parallelism and helps work that divides cleanly. Bigger workers buy memory and disk per task, which is what fixes spill, a Container killed by YARN for exceeding memory limits, a No space left on device from shuffle files, and a driver out-of-memory from listing too many files. If one task is the problem, a bigger worker is the move; if the stage is uniformly slow and shuffle-free, more workers is.

G.025X deserves its own note: it exists for streaming jobs whose volume is small but continuous, where the point is to run cheaply all day rather than to go fast.

AWS Glue developer guide — AWS Glue worker types

Glue job bookmark

The bookmark is the difference between an incremental job and one that reprocesses the world nightly, and what it stores depends entirely on the source — which is where its failure modes come from.

For Amazon S3 sources it compares the objects’ last-modified timestamps, not their names. So a file that is re-uploaded, copied, or touched by a lifecycle or re-encryption operation gets a new modification time and is processed again. Equally, a file back-dated into an already-processed window is never seen.

For JDBC sources it tracks bookmark keys — by default the table’s primary key — and compares values. The keys must be strictly monotonically increasing or decreasing; a column like a minor version or revision number whose values repeat across the dataset is not a valid key. And the decisive limitation: a JDBC bookmark captures only newly inserted rows. An UPDATE to an existing row does not advance any key, so it is invisible — which is precisely why CDC exists and why "use job bookmarks" is the wrong answer to a stem that mentions updates or deletes.

The documented ways a bookmark that is switched on still reprocesses everything, because they all look like a working job: the script omits job.init() or job.commit(), so the state is never initialised or never advanced; a transformation_ctx is missing or duplicated across frames, so there is no name to file the state under or two frames share one; the job reads with plain Spark DataFrames rather than DynamicFrames, which bypasses the Glue-side feature entirely; the JDBC bookmark keys are not sequential; or concurrent runs of the same job interleave their state. Every one of these leaves the output correct and the bill a multiple of what it should be.

Pause is the under-used third mode: process the increment without advancing the state, which is how you re-run a window for testing without losing your place. Disable is the default, so a job nobody configured is reprocessing the whole dataset by design rather than by accident.

AWS Glue developer guide — tracking processed data using job bookmarks

DynamicFrame

A Spark DataFrame has a single schema, which means reading messy semi-structured data either requires a schema you supply or an inference pass that fails on the first record that disagrees. A DynamicFrame instead keeps a choice of types per field where records differ, and gives you transforms to resolve them deliberately — ResolveChoice (cast, project, or make a struct of both), Relationalize (flatten nested arrays into joinable tables), ApplyMapping, DropNullFields, UnnestFrame.

It is also the type that Glue features attach to: job bookmarks, the S3 file-grouping options that fix the small-files problem, and bounded execution all take their configuration through create_dynamic_frame.from_options or from_catalog.

Convert in either direction with toDF() and fromDF(), and expect to: the usual shape of a real Glue job is to read as a DynamicFrame (for the bookmark and the messy-schema handling), convert to a DataFrame for the Spark SQL transformations, and convert back to write. A job that reads with spark.read instead gets none of the Glue-side features, which is the quiet reason a bookmark "does not work".

AWS Glue developer guide — DynamicFrame class

Flex execution class

The standard execution class gives a job dedicated capacity and starts it promptly. Flex runs the same job on non-dedicated capacity: it may wait to start, and a worker may be reclaimed mid-run, in which case Glue retries. In exchange the DPU-hour rate is materially lower.

The fit is decided by one question — does anything downstream have a deadline? A nightly backfill, a one-off reprocessing, a non-urgent enrichment job, and most development runs are all fine on Flex. An hourly job feeding a dashboard, or anything in the critical path of a dependent workflow, is not, because an unpredictable start time breaks the schedule contract rather than merely slowing it. Flex also suits jobs with a generous retry budget, since interruption is a normal outcome rather than an error.

A sibling lever worth pairing it with in a cost answer: job auto scaling, which lets Glue add and remove workers during the run based on the stage’s actual parallelism, so you specify a maximum rather than a fixed count and stop paying for idle executors during the narrow stages.

AWS Glue developer guide — Flex execution class and job auto scaling

Amazon EMR

EMR is the answer when you need the framework, the version, or the tuning that a managed abstraction does not expose. You choose a release label (which pins Spark, Hadoop and the rest), the applications installed, the instance types for primary, core and task nodes, and every configuration classification you care about. Three deployment forms: EMR on EC2 (the classic cluster, persistent or transient), EMR Serverless (no cluster to size), and EMR on EKS (jobs as pods in your existing Kubernetes cluster).

The comparison with Glue is a recurring exam decision, and the honest axis is control versus overhead rather than scale — both scale. Choose Glue for serverless Spark ETL with the Data Catalog, bookmarks and no capacity to manage. Choose EMR when the workload needs a non-Spark framework (Trino for interactive SQL, HBase for a wide-column store, Hive for an existing estate), a specific Spark or Hadoop version, custom JARs and native libraries at the cluster level, notebook-driven interactive work at scale, or deep per-cluster tuning. Choose EMR for cost too, at sustained volume: a long-running cluster on Reserved or Spot capacity beats per-DPU-second pricing once utilisation is high.

The cost mechanics to remember: task nodes hold no HDFS data, so they are safe to run on Spot and are where elasticity belongs, while core nodes carry HDFS and losing one loses data. A transient cluster that terminates on step completion is the cheapest way to run a scheduled batch job on EMR.

Amazon EMR management guide

EMR Serverless

You create an application pinned to a release and a framework (Spark or Hive), submit jobs to it, and EMR allocates workers for the duration of each job. Capacity is expressed as maximum vCPU and memory rather than as instance counts, and you are billed per worker-second for what ran. Pre-initialised capacity keeps a pool of warm workers so jobs start in seconds instead of waiting for provisioning, which is the knob that makes it viable for interactive or frequent short jobs — and which you pay for while it is warm.

Where it sits between its siblings. Against an EMR cluster: no sizing, no idle, no scaling policy, but also no cluster-level customisation, no HBase or Presto-style long-running service, and no SSH onto a node. Against Glue: you keep the EMR framework versions and configuration surface and the Spark-native experience, but give up the Glue-specific features — bookmarks, DynamicFrames, Glue Studio, the visual editor.

In a scenario, the signal for EMR Serverless is "we already run Spark on EMR and want to stop managing the cluster" or "spiky, intermittent batch workloads"; the signal against it is any requirement for a persistent service on the cluster.

Amazon EMR Serverless user guide

EMR on EKS

The EMR runtime, release versions and job-submission API, with Kubernetes as the resource manager instead of YARN. A virtual cluster maps to an EKS namespace; jobs become pods; the driver and executors are scheduled by Kubernetes alongside whatever else runs there.

Its whole value proposition is consolidation, and that is the signal to look for in a stem. An organisation that already runs EKS gets one compute platform, one set of node groups, one autoscaler, one capacity reservation strategy and one security model for both its services and its data jobs — rather than a separate EMR cluster sitting partly idle. Multi-tenancy is per namespace, so teams can be isolated with Kubernetes RBAC and IAM roles for service accounts.

The cost is Kubernetes expertise: scheduling, node groups, pod resource requests and the Spark-on-Kubernetes operational model are all now yours. A team with no EKS footprint should not acquire one to run Spark — that stem points at EMR Serverless or Glue.

Amazon EMR on EKS development guide

EMRFS and HDFS

HDFS on an EMR cluster lives on the core nodes’ volumes. It is fast — local disk, data locality — and ephemeral: terminate the cluster and it is gone. EMRFS implements the Hadoop file system interface over Amazon S3, so s3:// paths work everywhere an hdfs:// path would, with S3 durability, independent scaling, and access from any other engine.

The pattern this produces is the one the exam expects: S3 is the source of truth and HDFS is scratch. Read input from S3, let shuffle and intermediate stages use HDFS or local disk where the locality pays, write results back to S3, and let the cluster be disposable. A design that keeps the authoritative dataset in HDFS has coupled its data’s lifetime to a cluster’s, which is why "we must keep the cluster running to keep the data" is always a flag in a stem.

Two specifics worth carrying. S3 has been strongly read-after-write consistent for all operations since December 2020, which retired EMRFS consistent view — if you meet that feature in old material, it is no longer needed. And a rename on S3 is a copy plus a delete rather than a metadata operation, which is why committing many output files to S3 is slower than to HDFS and why optimised committers exist.

Amazon EMR management guide — work with storage and file systems

EMR managed scaling and instance fleets

Two independent levers that a cost question usually wants together.

Managed scaling watches YARN metrics and resizes core and task nodes between a minimum and maximum you declare, so a cluster shrinks when the queue empties instead of running at peak size all night. It replaces hand-written custom scaling policies for most workloads.

Instance fleets change how capacity is requested. Instead of naming one instance type per node group, you give a list of acceptable types with weighted capacities and a target in units for On-Demand and for Spot, plus an allocation strategy. EMR then fills the target from whatever is available and cheapest. This is what makes Spot genuinely usable: a fleet with five acceptable types rarely fails to find capacity, where a fleet of one type fails whenever that type is tight.

The rule that ties them to the cluster’s layout: put Spot on task nodes. Task nodes run no HDFS DataNode, so reclaiming one costs you in-flight tasks, which Spark retries. A core node carries HDFS blocks, so losing it can lose data and will at least trigger re-replication. Primary nodes on Spot risk the whole cluster.

Amazon EMR management guide — managed scaling and instance fleets

Apache ORC

ORC stores data in stripes (default around 64 MB), each holding column chunks plus an index with min/max values and optional bloom filters, and a footer carrying the schema and statistics. A reader can therefore skip whole stripes that cannot match a predicate and read only the columns a query references — the same two savings Parquet provides by the same means.

The real-world distinction is ecosystem rather than capability: ORC came out of the Hive world and is often marginally better on Hive and Trino workloads; Parquet came out of the Impala/Spark world and has the broader tool support, which is why it is the default answer on AWS and the one Firehose and Glue reach for. Both are splittable, both carry their schema, both support predicate pushdown. If a question offers ORC and Parquet as alternatives with no other distinguishing clause, it is almost certainly not testing the choice between them — it is testing columnar versus row-oriented.

Apache Avro

Avro is the one that is not columnar, and that is the point rather than a shortcoming. It serialises whole records compactly, with the writer’s schema available to the reader, and defines explicit compatibility rules for evolving that schema — add a field with a default and old readers still work; remove one with a default and new readers still work. Those rules are what a schema registry enforces.

So the shape of the answer is: Avro for write-heavy and streaming paths, Parquet or ORC for read-heavy analytical ones. Writing a single record to Avro costs one record’s work; writing it to Parquet means building column chunks and statistics, which is why streaming sinks buffer before converting. Reading one whole record from Avro is cheap; scanning one column across a billion rows is not, because every row must be traversed.

In an AWS pipeline you will often see both: Avro (or JSON) on the stream and in the raw landing zone, converted to Parquet in the curated zone. A stem that mentions frequent schema changes on an ingestion path, or a Kafka estate with a schema registry, is pointing at Avro; one that mentions analysts querying a wide table is pointing at Parquet.

Splittable compression

Spark, Athena and Hive parallelise by handing each task a byte range of a file. That only works if a reader can begin decoding at an arbitrary offset. Gzip cannot — the stream is one continuous DEFLATE sequence — so a 20 GB .csv.gz object is read by exactly one task, on one core, however large your cluster is. The job is not broken; it is serial, and the symptom is a stage with one task that runs for hours while everything else idles.

What is splittable: bzip2 (slow but splittable), LZO when indexed, and — importantly — Snappy or gzip inside Parquet or ORC, because the container is splittable even though each block’s codec is not. That last point resolves the apparent paradox that "gzip is not splittable" and "gzip-compressed Parquet is fine" are both true: the split happens at row-group or stripe boundaries, and each is decompressed independently.

The practical rules. Many moderate gzip files parallelise fine, because the unit of parallelism becomes the file — so 200 × 100 MB gzip objects are far better than one 20 GB object with the same bytes. Prefer a columnar format with Snappy for anything analytical. And reserve bzip2 for cases where a single large text file genuinely must be compressed and split.

Amazon Athena user guide — compression support, and Amazon Athena pricing examples

Compaction

Streaming ingestion and frequent micro-batches produce small files by construction — a buffer flushes every minute, so there is a file every minute. Compaction is the periodic job that reads a partition and writes it back as a handful of right-sized objects, after which queries over it are dramatically faster for exactly the reasons in the small-files problem.

How it is done on AWS. A Glue or Spark job reading a partition and writing with a controlled output count (coalesce or repartition before the write) is the general answer. In Spark, coalesce(n) reduces partitions without a shuffle by merging adjacent ones and is the cheap choice when reducing; repartition(n) shuffles to get an even distribution and is what you need when increasing the count or when the existing partitions are skewed. For an Apache Iceberg table the operation is first-class — a rewrite_data_files maintenance procedure, plus expiring old snapshots so the rewritten files can actually be deleted.

Two cautions. Compaction competes with readers for the same prefix, so it belongs on a schedule rather than in the ingestion path. And rewriting files changes their last-modified times, which is exactly what an S3 job bookmark keys on — a downstream incremental job will reprocess everything the compaction touched, which is the sort of interaction that only shows up in the bill.

Partition pruning

A Hive-style layout writes s3://lake/events/dt=2026-10-21/region=eu/part-0000.parquet, and the partition values live in the path, not in the file. When a query says WHERE dt = DATE 2026-10-21, the planner consults the Data Catalog (or partition projection) for the partitions that match and reads only those prefixes. Everything else is not scanned, not transferred, and not billed — which on Athena at $5 per TB scanned is the difference between cents and dollars per query.

The mechanism explains how it is defeated, and each of these is a plausible exam stem. Pruning needs the partition column in the predicate: filtering on an event_timestamp column inside the file does nothing for pruning even though it looks equivalent to filtering on dt. Applying a function to the partition column can prevent it — WHERE CAST(dt AS DATE) = ... or WHERE substr(dt,1,7) = ... may force a full enumeration where a literal comparison would not. New partitions must be registered: a crawler run, MSCK REPAIR TABLE, ALTER TABLE ADD PARTITION, or projection — unregistered data is simply invisible, which is the "we wrote the files but the query returns nothing" symptom. And partition columns must not also be stored as data columns in the file, or you have paid to store the same value twice.

Amazon Athena user guide — partitioning data, and Amazon Athena pricing

Over-partitioning

Partitioning is a trade, and both ends of it hurt. Too few partitions and every query scans too much. Too many and you have recreated the small-files problem with a metadata problem on top — the Catalog must enumerate and return hundreds of thousands of partitions during planning, which makes even a selective query slow to start, and each partition holds files too small to read efficiently.

The canonical mistake is partitioning by a high-cardinality column: user_id, an order id, or a timestamp down to the second. Partitioning hourly when every query filters by day is the subtler version — 24× the partitions for no pruning benefit.

The heuristics worth carrying. Partition on columns that appear in WHERE clauses, in descending order of how often they do. Aim for partitions that hold at least a few hundred megabytes. Use the coarsest granularity your queries actually filter on. And for a dated layout where the partition count is large but entirely predictable, partition projection removes the metadata cost by computing partition values from the key pattern instead of listing them — which is why it is the standard answer for log tables.

DAG (directed acyclic graph)

Acyclic is the load-bearing word. Because no path leads back to its own start, there is always at least one valid execution order, every task has a well-defined set of upstream dependencies, and the scheduler can run independent branches in parallel without a deadlock. A pipeline with a cycle has no such order, which is why every orchestrator rejects one.

The vocabulary is shared across the tools even where the syntax is not. Apache Airflow and Glue workflows build a DAG explicitly; Step Functions expresses one as a state machine whose Next fields are the edges; even Spark builds a DAG of stages internally. What differs is how a failure is expressed — which is the thing to compare when choosing, because every orchestrator can run A then B.

Two properties worth naming because questions turn on them. Fan-in (a task waiting on several upstreams) is where most real complexity lives, and where a tool’s join semantics matter. And a DAG describes dependency, not schedule: the same graph can be triggered hourly, on an event, or by hand, and conflating the two is how people end up with a scheduled job that runs before its input exists.

Express workflow

The two Step Functions workflow types differ in more than speed, and the differences decide which one a scenario needs.

Standard: up to one year per execution, exactly-once execution semantics, full execution history retained and visible in the console, priced per state transition, and it supports the human-approval and callback patterns via a task token. This is the data-pipeline default — a nightly ETL chain with twelve steps is a handful of transitions and costs almost nothing.

Express: up to 5 minutes, at-least-once semantics, no retained execution history in the console (logs go to CloudWatch Logs, which you must enable), priced per request and duration, and capable of very high invocation rates. Synchronous Express executions return a result to the caller; asynchronous ones do not.

The discriminators to listen for. "Thousands of executions per second" or "per-event processing" → Express. "The workflow runs for two hours" or "a human approves step three" or "it must not run twice" → Standard. The cost reasoning inverts with volume: Standard is cheap for a few long workflows and ruinous for millions of short ones, which is exactly the shape Express exists for.

AWS Step Functions developer guide — standard vs express workflows

Distributed Map

The Map state has two modes and the gap between them is large. Inline mode runs iterations inside the parent execution: maximum 40 concurrent iterations, input must be a JSON array, and every iteration’s events are appended to the parent’s history. Distributed mode runs each iteration as a separate child workflow execution with its own history, giving up to 10,000 parallel child executions and the ability to take an Amazon S3 object listing, a CSV file, a JSON file, or an Athena manifest as the item source.

AWS states the switch conditions plainly, and they are the exam’s decision rule: use Distributed mode when the dataset exceeds 256 KiB, when the execution history would exceed 25,000 entries, or when you need more than 40 concurrent iterations. All three are the same underlying limit of the parent execution being asked to carry everything.

This is the state that makes Step Functions a credible data-processing orchestrator rather than only a coordinator: "process every object under this prefix" becomes one state with a maxConcurrency instead of a hand-written fan-out. Set that concurrency deliberately — the usual mistake is launching 10,000 children against a downstream service, or a Lambda account concurrency, that can absorb 1,000.

AWS Step Functions developer guide — Map state processing modes

Task token (callback pattern)

Invoked with the .waitForTaskToken integration pattern. The state machine passes a token to the target, the state pauses — not polling, not billed for waiting — and resumes only when something calls SendTaskSuccess or SendTaskFailure with that token. A heartbeat timeout and a task timeout bound the wait.

Where it earns its place in a data pipeline: a manual approval before a destructive backfill; an on-premises or third-party system that must do a step and report back; a long-running job on a platform Step Functions has no optimised integration for. It is the clean alternative to the anti-pattern of a Lambda function that sleeps, or a Wait loop that polls every thirty seconds for four hours and bills a state transition each time.

The constraint that makes it an exam discriminator: callbacks are a Standard-workflow feature. An Express workflow cannot wait on a task token, and it could not usefully anyway — its ceiling is five minutes. A stem combining "human approval" with "Express" is self-contradictory, and noticing that is often the whole question.

AWS Step Functions developer guide — wait for a callback with the task token

Amazon MWAA

You choose an environment class and an Airflow version; AWS runs the scheduler, the metadata database, the web server and an auto-scaling worker pool. Your DAG files, plugins.zip and requirements.txt live in an S3 bucket that MWAA synchronises, so deployment is an S3 upload — which is also why a bad requirements.txt can break the whole environment rather than one DAG.

Why a team chooses it over Step Functions, honestly stated. Python-defined DAGs with real control flow and shared helper code. The Airflow operator and provider ecosystem, which covers hundreds of third-party systems with no integration code. Sensors and the data-interval model, which make dependency on external state and on time windows natural. Backfill as a first-class operation. And portability: the DAGs run on any Airflow.

What it costs. The environment is billed while it exists, not per execution, so an idle MWAA environment has a floor that a Step Functions state machine does not — which is the deciding factor for a small pipeline. You own Airflow’s own operational surface: version upgrades, a scheduler that can fall behind, worker sizing, and the metadata database growing until someone prunes it. The exam’s shorthand is fair: existing Airflow DAGs or a complex, Python-heavy, many-source pipeline → MWAA; a handful of AWS services to sequence with retries → Step Functions.

Amazon Managed Workflows for Apache Airflow user guide

Airflow operator

An operator is the unit a DAG is assembled from. Instantiating GlueJobOperator(job_name=...) creates one task; the DAG is the operators plus their dependency edges. The AWS provider package ships operators and sensors for Glue, EMR, EMR Serverless, Athena, Redshift, Step Functions, Lambda, S3, SageMaker and more, so the integration code a hand-rolled orchestrator would need is already written and tested.

Two distinctions worth holding. An operator does something; a sensor waits for something. And a deferrable operator releases its worker slot while waiting, handing the wait to a lightweight triggerer process — which matters because the naive form of "wait four hours for a file" occupies a worker slot for four hours, and a DAG full of those starves the pool while the scheduler reports everything as running.

The design rule that follows from the retry model: a task should be idempotent, because Airflow will retry it. An operator that appends to a table will append twice on a retry. See idempotent write.

Airflow sensor

Sensors are how a DAG depends on something it does not control. S3KeySensor waits for an object; GlueCatalogPartitionSensor waits for a partition to be registered; ExternalTaskSensor waits for a task in a different DAG. Without them, a scheduled pipeline has to assume its input arrived, which is the assumption that produces an empty output and a green run.

The operational hazard is mode, and it is the detail an exam question about a starved scheduler is built on. The default poke mode holds a worker slot for the entire wait. reschedule mode releases the slot between checks and is the right default for a long wait. A deferrable sensor goes further and hands the wait to the triggerer entirely. Every sensor also needs a timeout, or a condition that never becomes true leaves a task running until someone notices.

The design alternative worth naming: a sensor polling for a file is a pull; an S3 event notification triggering the DAG is a push. The push has lower latency and no polling cost, and the pull is simpler when several inputs must all be present before anything may start — which is precisely the fan-in case sensors are good at.

Glue workflow and triggers

A Glue workflow holds jobs and crawlers as nodes and triggers as edges. A trigger is scheduled (cron), on-demand, or conditional — fire when a listed set of jobs and crawlers has reached a listed set of states, with AND or ANY semantics, which is how fan-in is expressed. Workflow-level run properties pass values between nodes, and the console shows a run graph with per-node status.

Its proper place is small: a pipeline that is entirely Glue jobs and crawlers and needs no other service. For that case it is the lowest-overhead option available, with nothing extra to provision and no separate IAM surface. The moment the pipeline must call Lambda, run an EMR step, wait for a human, branch on a computed value, or do anything conditional beyond job states, it has outgrown Glue workflows and belongs in Step Functions or MWAA.

For a question, the signal is the inventory of steps. All-Glue, linear or simple fan-in → Glue workflow. Mixed services or real branching → something else. There is no cost argument either way, which is why the answer turns purely on fit.

AWS Glue developer guide — workflows and triggers

EventBridge Scheduler

Distinct from an EventBridge rule with a schedule, which is the older mechanism. Scheduler is built for scheduling at scale: millions of schedules, a flexible time window to spread load rather than firing everything exactly on the minute, built-in retry with a maximum age, a dead-letter queue per schedule, and — the one people most often need — time zone support with automatic daylight-saving adjustment, where a cron rule is UTC and silently shifts an hour twice a year relative to local business time.

It also supports one-time schedules (at(2026-12-01T03:00:00)), which is how you schedule a single future action without inventing a cron expression you then have to remember to delete.

In a data pipeline it is the trigger at the top of a DAG — start a Step Functions execution, a Glue job, a Lambda — and the thing to reach for when a stem complains that a nightly job ran at the wrong local hour after a clock change, or that thousands of schedules all firing at midnight overwhelm a downstream service.

Amazon EventBridge Scheduler user guide

EventBridge Pipes

A pipe has four stages: a source (SQS, Kinesis, DynamoDB Streams, MSK, Amazon MQ), an optional filter (an event pattern evaluated before anything is billed downstream), an optional enrichment (Lambda, Step Functions, API destination, API Gateway), and a target (any of 20-odd AWS services). Polling, batching, retries, ordering and failure destinations are the service’s problem.

What it replaces is a very recognisable piece of code: the Lambda function whose whole body is "receive from the stream, drop the records I do not care about, reshape the rest, call the next service". That function has to be written, tested, deployed, monitored and paid for per invocation, and the filter runs after the invocation — so you are billed for the records you discard. A pipe filters before the target is engaged.

The comparison to keep straight: an EventBridge bus is one-to-many routing with rules fanning one event to several targets; a pipe is one source to one target with transformation in between. A stem about fanning one event out to four consumers is a bus; one about moving a DynamoDB change feed into a Firehose stream with a filter is a pipe.

Amazon EventBridge user guide — EventBridge Pipes

S3 event notification

Configured per bucket with an event type (s3:ObjectCreated:*, s3:ObjectRemoved:*, and others) and optional prefix and suffix filters, so only raw/ objects ending .parquet fire. Delivery is at-least-once, which means a duplicate notification for one object is normal and the handler must be idempotent.

Two routes, and the exam distinguishes them. Direct notifications to SNS, SQS or Lambda are the classic path: simple, low latency, and limited — the filters are prefix and suffix only, and historically one overlapping-prefix configuration per bucket was a real constraint. EventBridge integration, enabled with one setting on the bucket, publishes every S3 event to the default bus instead, where rules can match on any field of the event (size, object key pattern, requester), fan one event out to several targets, be archived and replayed, and be routed cross-account. For anything beyond "call this one function", EventBridge is the better answer.

The pattern to recognise for a high-volume landing zone: S3 → SQS → a consumer. The queue absorbs bursts, gives a retry and a dead-letter queue for free, and decouples the arrival rate from the processing rate — where S3 → Lambda directly will happily try to invoke ten thousand concurrent functions and meet your account concurrency limit instead.

Amazon S3 user guide — Amazon S3 Event Notifications

Window function

The distinction from GROUP BY is the whole point: a group-by returns one row per group, a window function returns every row with the aggregate attached. That is what lets you rank within a category, compute a running total, compare a row to the group average, or take the previous value with LAG.

The form a data engineer writes most, and the one that appears in exam stems, is deduplication:

SELECT * FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) rn FROM t) WHERE rn = 1

Read it as: group the rows by id, order each group newest first, number them, keep the first of each. That is "the latest version of every record", which is exactly what you need over an append-only change feed — see applying change events downstream.

The performance note that matters, because it decides whether the query is viable. PARTITION BY is a shuffle key: the window function is a wide transformation, so a window over a high-cardinality key is a full shuffle of the dataset, and one over a skewed key is a skew problem. ROW_NUMBER, RANK and DENSE_RANK also differ on ties — RANK leaves gaps, DENSE_RANK does not, ROW_NUMBER breaks ties arbitrarily — and picking the wrong one silently changes a deduplication result.

Common table expression (CTE)

WITH recent AS (SELECT ...) SELECT ... FROM recent JOIN .... A CTE names an intermediate result so a complex statement reads as a sequence of steps instead of nested subqueries, and it can be referenced more than once in the same statement.

The misconception worth correcting, because it generates real cost surprises: a CTE is not a temporary table and carries no guarantee of being computed once. Engines are free to inline it into each reference, so a CTE referenced three times may scan its source three times. If you need it computed once, materialise it deliberately — a CREATE TABLE AS SELECT in Athena, a temporary table in Redshift, a .cache() in Spark — and accept the storage.

Recursive CTEs (WITH RECURSIVE) are the other half of the term and the one a stem about hierarchies points at: walking an org chart, a bill of materials, or a parent-child category tree to arbitrary depth. They are supported in Redshift and in Trino-based Athena engines; they are also the easiest way to write a query that never terminates, so a depth guard is not optional.

Query execution plan

The plan is the only way to answer "why is this query slow" without guessing, and it answers it by naming the one or two operations that dominate. What to look for, in the order it pays off:

Join strategy. A broadcast or hash join against a small side is cheap; a sort-merge or shuffle join against two large sides is the expensive node, and seeing it tells you whether a broadcast is available. Redshift’s plan says this as DS_BCAST_INNER (broadcasting the inner table to every node) or DS_DIST_BOTH (redistributing both — the expensive one) versus DS_DIST_NONE, which means the data was already co-located by distribution key and nothing moved.

What was pushed down. A plan that shows the partition filter and the column list at the scan node is pruning; one that shows a full scan with the filter applied above it is not — see partition pruning and pushdown.

Row estimates versus reality. EXPLAIN ANALYZE (Athena, Trino, Redshift’s EXPLAIN with statistics) shows both, and a plan whose estimate is wrong by orders of magnitude is a plan built on missing statistics — which is the real fix rather than a hint.

In Spark the equivalent surface is the SQL tab of the Spark UI plus df.explain(True), where the thing to find is the Exchange nodes: each one is a shuffle, and their number is the first-order cost of the job.

Change set

Submitting a template directly applies it. Creating a change set instead produces a list of resource-level actions with no changes made, which you then execute or discard. The column to read is Replacement: True means CloudFormation will create a new resource and delete the old one, which for a database, an EBS volume or anything stateful means the data is gone. That is the whole reason change sets exist, and reviewing them is the practice that separates a routine deployment from a Friday-evening incident.

The neighbouring concepts a question will test alongside it. Drift detection compares the deployed resources with the template and reports what someone changed by hand — a template is only the truth if nobody edited the console. Stack policies deny update actions on named resources, which is a hard guard where review is a soft one. StackSets deploy one template across many accounts and Regions from a single operation. And a deletion policy of Retain or Snapshot on a stateful resource is what prevents a stack deletion from taking the data with it.

For a data pipeline the stateful resources to protect are the obvious ones — an S3 bucket, a Glue Catalog database, a Redshift cluster, a Kinesis stream — and the property changes that force replacement on each are worth knowing before, not during, a release.

AWS CloudFormation user guide — updating stacks using change sets

AWS SAM

A SAM template is a CloudFormation template with Transform: AWS::Serverless-2016-10-31 and a handful of condensed resource types — AWS::Serverless::Function, ::Api, ::StateMachine — that expand into the dozen underlying resources (function, role, policies, log group, event source mapping, permissions) CloudFormation would otherwise need spelled out. It is CloudFormation underneath, so change sets, drift detection and stack policies all still apply.

The part that earns it a place in a pipeline toolkit is the CLI: sam build handles dependency packaging per runtime, sam deploy --guided manages the deployment bucket and parameters, and sam local invoke / sam local start-api run the function in a container with a synthesised event so you can test a Lambda transform without deploying it.

Where it sits among the alternatives, which is the comparison a stem wants: raw CloudFormation for arbitrary infrastructure in YAML or JSON; CDK when you want a real programming language, loops, conditionals and shared constructs; SAM when the stack is mostly serverless and the local-testing and packaging workflow is the point; Terraform when the estate is multi-cloud or the team already standardised on it.

AWS Serverless Application Model developer guide

Trunk-based development

The alternative — a branch per feature, living for weeks — accumulates divergence, and the cost of a merge grows with the square of the time since the branches parted. Trunk-based development keeps main always releasable, merges small changes continuously, and hides unfinished work behind a flag rather than behind a branch. GitFlow is the named counter-example, with its long-lived develop, release and hotfix branches, and it suits versioned products shipped to customers far better than it suits a continuously deployed pipeline.

Why a data engineer is asked about it at all: a pipeline’s code and its infrastructure definition evolve together, and a long-lived branch of infrastructure code is a stack that has drifted from production in ways nobody has tested. The pattern that goes with trunk-based development here is environment promotion — the same template or CDK app deployed to dev, then staging, then production, with parameters as the only difference — so "it worked in staging" means something.

The data-specific wrinkle worth stating: a feature flag can hide a code path, but it cannot hide a schema change. Those need the expand-and-contract discipline instead: add the new column, write both, migrate readers, then stop writing the old one — three deployments, never one.

Paginator

Almost every AWS list or describe API is paginated: ListObjectsV2 returns up to 1,000 keys, GetPartitions and Scan return a page plus a token. Calling one once and iterating the result is a bug that looks like working code — it returns data, no error is raised, and it is wrong by however much did not fit in one page. This is one of the most common real defects in hand-written data tooling, and it is why the construct appears in exam stems.

The SDK’s answer is a paginator: paginator = client.get_paginator("list_objects_v2") then iterating paginator.paginate(Bucket=..., Prefix=...), which issues the follow-up calls and yields every page. The manual equivalent is a loop on NextToken / NextContinuationToken / LastEvaluatedKey until it is absent — and getting the terminating condition wrong in either direction gives you a partial result or an infinite loop.

Two companions in the same family. Waiters poll a describe API until a resource reaches a state (cluster_available, object_exists) with sensible backoff, which is the correct replacement for a hand-written sleep loop. And the batch APIs — PutRecords, BatchWriteItem — return partial failures inside a successful response, so the loop that re-submits unprocessed items is as necessary as the paginator and omitted just as often.

AWS SDK for Python (Boto3) documentation — paginators and waiters

Access pattern

The input to every store decision, and the thing most designs skip. A usable access pattern names four things: the predicate ("by customer id", "by a date range plus a status"), the selectivity (one row out of a billion, or a hundred million rows aggregated), the latency budget (single-digit milliseconds for a checkout page, seconds for a dashboard, minutes for a nightly report), and the write shape (one row at a time by a key, or a million rows appended in a batch). Two systems with identical data and different access patterns belong in different stores. The reverse mistake is more common: picking the store first and then discovering the pattern it cannot serve.

Selectivity

A point lookup of one order out of two billion is extremely selective; a monthly revenue total over the same table is not selective at all. The distinction decides the storage engine, because the two are served by opposite machinery: selective reads want a key-ordered structure that can jump straight to the row, and unselective reads want a columnar layout that streams one column of everything at maximum throughput. A store tuned for one is usually bad at the other, which is why a single table rarely serves both an application and an analyst without a second copy.

S3 prefix

An object key is one flat string; the slashes in year=2026/month=03/part-0.parquet are ordinary characters that consoles and query engines choose to render as a tree. Two consequences matter. Listing is a scan of a sorted key space, so ListObjectsV2 with a prefix is cheap and listing a bucket to find one file is not. And request rate is per prefix: Amazon S3 supports at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes — so a job that throttles is usually a job whose keys all share one prefix, and spreading the key space is the fix.

Amazon S3 User Guide — optimizing Amazon S3 performance

Columnar format

Apache Parquet and Apache ORC store values column by column inside row groups, with per-column statistics (minimum, maximum, null count) in the footer. Three wins follow, and all three are billable on Athena. Projection pushdown: naming three of forty columns reads roughly three-fortieths of the bytes. Predicate pushdown: a row group whose maximum value for order_date is below the filter is skipped without decompression. Compression: like values sit next to each other, so dictionary and run-length encodings do far better than they can on interleaved rows. Row formats — CSV, JSON, Avro — have none of this, which is why the standard lake pattern is to land raw in whatever arrives and convert to columnar before anyone queries it.

Small-file problem

Every file a query engine touches costs at least one GetObject, a round trip of latency, a footer read, and a split in the planner. A table of 200,000 files of 50 KB each holds the same 10 GB as 20 files of 500 MB, but the first scans with hundreds of thousands of sequential small reads and the second saturates available bandwidth. Streaming ingestion is the usual cause — a Firehose or Spark micro-batch writing every few seconds produces exactly this shape. The fixes are buffering at write time, a compaction job, or an open table format with managed compaction. The exam reaches for it whenever a stem says "query performance degraded as the table grew" without mentioning partitions.

Amazon S3 Tables

A third bucket type alongside general purpose and directory buckets, addressed through the s3tables API, holding namespaces and tables rather than free-form keys. The value is that the three maintenance jobs every self-managed Iceberg table needs are built in and configurable: compaction (default target file size 512 MB, settable 64–512 MiB, strategy auto/binpack/sort/z-order), snapshot management (keep at least 1 snapshot, maximum snapshot age 120 hours by default) and unreferenced file removal (unreferenced for 3 days, then deleted after 10 noncurrent days). Those deletions are permanent, so the retention defaults are a real design decision rather than a detail — a 120-hour snapshot window is also your time-travel window.

Amazon S3 User Guide — S3 Tables maintenance

S3 Express One Zone

Lives in a directory bucket, not a general purpose bucket, and co-locates the data with your compute in one Availability Zone to cut request latency by an order of magnitude — the right store for the shuffle and spill traffic of an interactive Spark or query workload reading the same objects thousands of times. Three constraints decide exam questions: it is one zone, so the data is lost if that zone is lost and it needs a durable copy elsewhere; storage is priced well above S3 Standard while requests are priced well below, so it only pays on genuinely hot data; and directory buckets do not support everything general purpose buckets do, so it is not a drop-in replacement for a lake’s landing zone.

Amazon Redshift

A leader node parses and plans; compute nodes hold the data and execute in parallel across slices, each slice an independent unit of work with its own portion of the table. Everything distinctive follows from that shape: a join is fast when the two sides’ matching rows already sit on the same slice (see distribution style) and slow when the planner has to redistribute or broadcast them first. Storage is columnar and compressed per column, with zone maps letting the scan skip whole blocks. Deployment comes in two forms — provisioned clusters on RA3 nodes with Redshift Managed Storage, and Redshift Serverless billed in RPU-seconds for workloads too spiky to size.

Amazon Redshift Database Developer Guide

Distribution style

KEY hashes one column and places equal values on the same slice, so two tables distributed on their join column are collocated and joined with no network traffic. ALL puts a full copy of the table on every node — right for a small dimension that joins everything, and it multiplies storage by the node count and slows loads. EVEN round-robins rows regardless of value, which is correct for a table that does not join. AUTO is the default and is managed by automatic table optimization: small tables start as AUTO(ALL), large ones as AUTO(EVEN), and Redshift may promote a table to KEY on the primary key as it grows. A fact table has only one distribution key, so collocating with a second dimension is not an option — that is the constraint behind most distribution questions.

Amazon Redshift Database Developer Guide — distribution styles

Distribution key (DISTKEY)

Choose it for the join you actually run: designate the fact table’s foreign key and the dimension table’s primary key as DISTKEY and their rows collocate. Two traps follow from the hash. Low cardinality concentrates rows — distributing on a status column with four values puts the whole table on four slices and leaves the rest idle. Cardinality has to hold after filtering, not just overall: a sales table distributed on date looks evenly spread, but a query restricted to one narrow date range reads from a handful of slices while the others wait, which is skew that only appears under the real workload.

Sort key (SORTKEY)

Sorting is what makes a range-restricted predicate cheap: the zone map holds the minimum and maximum of the sort column per block, and any block whose range cannot satisfy the filter is never fetched. That makes the column you filter on most — almost always a timestamp or date on a fact table — the first sort key. A compound sort key sorts by the listed columns in order and only helps when a query filters on the leading column; an interleaved sort key gives equal weight to each column, which suits unpredictable filtering but costs more to maintain. Rows loaded out of order become unsorted and need VACUUM; automatic table optimization will also propose and apply sort keys.

Slice

The unit that parallelism is actually measured in. A query finishes when its slowest slice finishes, which is why uneven data distribution — skew — costs more than it looks: one slice holding ten times its share does ten times the work while the rest idle. Loading data in a number of files that is a multiple of the slice count lets every slice pull in parallel; a single large uncompressed file is read by one slice and is the classic slow COPY.

Redshift Managed Storage (RMS)

On RA3 node types and on Redshift Serverless, a table’s durable home is managed storage backed by Amazon S3, with local SSD acting as a cache for frequently accessed blocks. The design consequence is the one the exam cares about: you size compute for your query workload and let storage grow on its own, instead of adding nodes you do not need in order to get disk. It is also what makes data sharing and fast snapshot restore practical, because a snapshot is a reference to blocks already in managed storage rather than a bulk copy.

Amazon Redshift Serverless

You create a namespace (the database, its users and its data) and a workgroup (the compute and network settings), set a base capacity in Redshift Processing Units and optionally a maximum, and pay for the seconds the workgroup is actually serving queries. It suits spiky, intermittent or unpredictable analytics — a dashboard used in business hours, an ad-hoc analyst population, a development environment — where a provisioned cluster would be idle most of the day. A steady, round-the-clock workload with a known shape is usually cheaper on provisioned RA3 nodes with a reserved-instance commitment, which is the trade-off a cost question is testing.

Amazon Redshift Spectrum

An external schema maps a Redshift database to a catalog — usually the AWS Glue Data Catalog — and external tables over S3 files can then be joined to local Redshift tables in one query. Scans are pushed down to the Spectrum fleet and billed per terabyte scanned, so the same rules that govern Athena cost apply: partition the external data, store it columnar, and project only the columns you need. The design pattern it enables is the one lifecycle questions reach for — keep the last two years in managed storage where joins are fast, leave eight years of history in S3, and define a view that unions the two so analysts see one table.

Concurrency scaling

Turned on per workload-management queue. When queries start queueing, Redshift routes the overflow to additional clusters with the same data, and the results are indistinguishable to the client. Each 24 hours of main-cluster uptime earns one hour of concurrency-scaling credit, so bursty read traffic is often free in practice and sustained overload is not. It is the correct answer when a stem says reports queue during a daily peak but the cluster is right-sized the rest of the day — and the wrong answer when the problem is one enormous query, which concurrency scaling does nothing for.

Workload management (WLM)

The mechanism that stops an analyst’s runaway scan from starving the nightly load. Automatic WLM lets Redshift set concurrency and memory itself and supports query priority, which is the recommended default; manual WLM lets you pin a number of slots and a memory percentage per queue. Query monitoring rules sit on top and act on measured behaviour — abort, hop to another queue, or log, when a query exceeds a threshold such as rows returned or execution time. The exam reaches for WLM whenever a question separates "an ETL job that must finish" from "ad-hoc queries that may wait".

Materialized view

In Redshift a materialized view holds the rows its query produced and can be refreshed manually, on a schedule, or automatically; many refreshes are incremental, reapplying only changed base-table rows rather than recomputing from scratch. Redshift can also rewrite an unrelated query to use a materialized view it did not name, so a dashboard benefits without being edited. They can be defined over external tables, which makes them a practical way to cache an expensive Spectrum scan. The cost is staleness: a materialized view answers as of its last refresh, so the design question is always what lag the consumer tolerates.

Zero-ETL integration

Available from Amazon Aurora MySQL and PostgreSQL, Amazon RDS for MySQL, Amazon DynamoDB and several third-party applications. AWS reads the source’s change stream and keeps a Redshift database in step within seconds, so analytics run on near-current operational data without an extract job. What it does not do is transform: the target mirrors the source’s shape, so modelling, conforming and history-keeping still happen downstream in Redshift. It is the right answer when a stem asks for analytics on operational data with minimal engineering effort, and the wrong one when the requirement includes reshaping or enriching the data in flight.

Amazon DynamoDB

Items live in partitions chosen by hashing the partition key; an optional sort key orders items that share one. The performance promise holds because the engine never searches: a Query must specify exactly one partition key value and may add a condition on the sort key, and anything else is a Scan of the whole table. Capacity is either provisioned (with auto scaling) or on-demand, which instantly absorbs up to double the previous peak. The hard ceiling the exam tests is per partition — roughly 3,000 read units or 1,000 write units per second — and no amount of table-level capacity raises it for a single hot key.

Amazon DynamoDB Developer Guide

Amazon Aurora

Aurora separates the database engine from storage. A cluster volume grows automatically and replicates every write six ways across three Availability Zones; read replicas attach to that same volume instead of replaying a log, which is why replica lag is typically milliseconds and why adding a reader costs no extra storage. Aurora Serverless v2 scales capacity in fine-grained Aurora Capacity Units in place, without a reconnect. For a data engineer it matters as a source: Aurora is where zero-ETL integrations to Redshift originate, and its cluster endpoints let an extract job read from a reader without touching the writer.

Amazon RDS

Chosen over Aurora when the engine or feature set requires it (Oracle and SQL Server are RDS-only), when an existing application expects exact upstream behaviour, or when the workload is small enough that Aurora’s floor costs more. Multi-AZ gives a synchronous standby for failover, not for reads; read replicas give reads and lag asynchronously. For pipelines, RDS is both a source — a logical-replication slot feeding AWS DMS or a zero-ETL integration — and the place a question puts a workload that someone is about to propose moving to Redshift for the wrong reason.

Read replica

The standard way to let reporting or extraction read a production database without competing with transactions. Two distinctions the exam leans on: a read replica is not a high-availability feature — that is Multi-AZ, whose standby serves no traffic — and replica lag means a read can return data the writer has already superseded, which breaks any extract that assumes it has seen every committed row at a point in time. When an extract must be exactly consistent, read a snapshot or use the source’s change stream rather than a replica.

Amazon ElastiCache

A cache is defined by what it is allowed to lose. ElastiCache sits in front of a database or an API to absorb repeated reads, and the application must be able to rebuild any entry from the source of truth — which is why a cache miss is a normal event rather than a failure. For pipelines it appears in two shapes: a lookup cache for a dimension a stream enrichment step hits thousands of times a second, and a dedupe or idempotency store holding recently seen record keys. If the data cannot be rebuilt, the design wants Amazon MemoryDB instead.

Amazon MemoryDB

Valkey- and Redis-compatible, with the same data structures and commands as ElastiCache, and one architectural difference that changes which questions it answers: every write is committed to a multi-Availability-Zone transaction log before it is acknowledged. That makes MemoryDB durable, so it can be the primary store rather than a copy of one. The trade-off is in the acknowledgement path — writes are slower than a pure cache’s (single-digit milliseconds against microseconds) while reads stay in microseconds — and in price per gigabyte, which is memory pricing.

Amazon Keyspaces

The reason to choose it over DynamoDB is almost always compatibility rather than capability — an application already speaks the Cassandra Query Language, uses its drivers, or depends on its wide-partition data model, and rewriting it is not on the table. Capacity is on-demand or provisioned, tables are replicated across three Availability Zones, and point-in-time recovery and time-to-live work much as they do in DynamoDB. For a greenfield key-value workload on AWS, DynamoDB is the default and Keyspaces is the migration answer.

Amazon DocumentDB

Separates compute from a storage layer that replicates six ways across three Availability Zones, with up to 15 read replicas, and emulates the MongoDB wire protocol so existing drivers and tools connect unchanged. Pick it for the API, not for the shape of the data: DynamoDB also stores documents, and the decision between them turns on whether the workload needs MongoDB’s richer ad-hoc query and secondary-index model or DynamoDB’s stricter key-addressed model and scaling behaviour.

Amazon Neptune

Supports the property-graph languages Gremlin and openCypher plus the RDF language SPARQL, and the discriminator in a store question is the unbounded traversal: "find every account connected to this one through any chain of shared devices" is a query whose depth is not known in advance. In SQL that is a recursive join whose cost explodes with depth; in a graph store, traversal is the primitive. Fraud rings, identity resolution, lineage and recommendation sit here. Aggregating over millions of rows does not — that is still the warehouse.

Amazon Timestream

Time-series stores organise data by time first and interpolate, downsample and window it natively. Two variants exist and their lifecycle status differs, which is the fact to carry into an exam: Timestream for LiveAnalytics — the serverless engine with automatic memory-to-magnetic tiering — was closed to new customer access effective 20 June 2025, with existing workloads unaffected and still supported; Timestream for InfluxDB is the managed InfluxDB offering AWS recommends for new time-series workloads. On a question written before that change, LiveAnalytics may still be the keyed answer; on a design decision made today it is not available to start with.

Amazon Timestream Developer Guide — availability change (20 June 2025)

Hive metastore

Hadoop-era engines needed somewhere to record that a directory of files is a table with named, typed columns, read by a particular serialisation library. The Hive metastore is that register, traditionally a relational database you ran yourself. The AWS Glue Data Catalog implements the same model as a managed, account-wide service, which is why Athena, Amazon EMR, Amazon Redshift Spectrum and Spark can all read the same table definition. Migrations matter in questions: Amazon EMR can be pointed at either, and a self-managed metastore can be imported into Glue, but two catalogs describing the same files is a drift problem waiting to happen.

AWS Glue crawler

It samples rather than reads everything. For CSV it reads the first 1,000 records or the first 1 MB, whichever comes first; for JSON the first 1 MB, extended in 1 MB steps to a maximum of 10 MB if no valid record has been found; for Parquet it takes the schema straight from the file footer. It then compares the schemas inferred across every file and folder and creates one table per distinct shape — so differing format, compression codec, schema or partition structure under a single path yields several tables rather than one. Its update behaviour on a later run is configurable, and it can be scheduled or triggered on an event.

AWS Glue Developer Guide — using crawlers to populate the Data Catalog

Classifier

A crawler runs its list of custom classifiers in the order you gave them, and the first one that returns certainty 1.0 wins; if none is certain, the highest-certainty result is used, and only then does the crawler fall back to the built-in classifiers for CSV, JSON, Parquet, ORC, Avro, XML and the common log formats. Writing a Grok, XML, JSON-path or CSV classifier is the fix when the built-in guess is wrong — most often a headerless or oddly delimited CSV where every column is inferred as a string and the first data row is consumed as column names.

CombineCompatibleSchemas

Set as {"Grouping": {"TableGroupingPolicy": "CombineCompatibleSchemas"}}, it tells the crawler to stop treating schema similarity as a reason to split: files that differ only by, say, an extra optional column collapse into one table definition. The limit, and the exam’s angle on it, is that the crawler still checks compatibility — genuinely incompatible data, such as the same column typed int in one file and string in another, still produces multiple tables however the policy is set. Fixing that needs the data corrected or the type widened, not a different crawler option.

AWS Glue Developer Guide — creating a single schema for each Amazon S3 include path

Crawler schema-change policy

Two independent settings. On a changed schema the crawler can UPDATE_IN_DATABASE — overwrite the table definition, which is convenient and will happily replace a hand-tuned schema — or LOG, which records the change and leaves the table alone. On data that has disappeared it can DELETE_FROM_DATABASE, DEPRECATE_IN_DATABASE (mark the table deprecated but keep it) or LOG. The combination worth knowing for a question about accidental loss is that DELETE_FROM_DATABASE removes catalog entries irreversibly when a prefix is temporarily empty, so DEPRECATE_IN_DATABASE is the safe default, and a crawler should not be pointed at a table whose schema is maintained deliberately.

Hive-style partitioning

A key of sales/year=2026/month=03/day=14/part-0.parquet tells any Hive-compatible engine that this file belongs to year=2026, month=03, day=14 without being told the column names separately. Keys written as sales/2026/03/14/ carry the same information positionally but not the names, so partitions have to be registered explicitly with ALTER TABLE ADD PARTITION or a storage.location.template under partition projection. The naming convention is therefore not cosmetic: it is the difference between MSCK REPAIR TABLE working and not working.

Partition (catalog)

Partition columns are not stored in the data files — their values come from the catalog entry or from the key itself — which is why selecting one costs nothing to read. The register has weight of its own, though: a table partitioned by day for ten years holds about 3,650 partition entries, by hour about 87,600, and by hour and customer, millions. Every query begins with a GetPartitions call against that register, and the planning time it takes grows with the count, which is the mechanism behind "the query got slower as the table grew" even when the bytes scanned did not change.

Partition index

Without an index the Data Catalog loads all of a table’s partitions and then applies the request’s filter expression, so planning time rises with the partition count regardless of how selective the query is. With an index it fetches only matching partitions. The rule that decides whether it is used at all: the first key of the index must appear in the expression. For an index on [country, category, year], a filter on country uses it and a filter on category alone does not. Optimisation is best-effort — an OR anywhere in the expression, or an unsupported operator, falls back to loading everything — and a table may carry up to three indexes for different query shapes.

AWS Glue Developer Guide — creating partition indexes

MSCK REPAIR TABLE

Useful for a one-off load of a table whose keys follow the column=value/ convention. It is the wrong steady-state tool for two reasons: it lists the whole prefix every time, so its runtime grows with the table and can time out on a large one, and it only adds partitions — it never removes entries whose data is gone. The maintainable alternatives are ALTER TABLE ADD PARTITION issued by the job that wrote the data, a scheduled crawler, or partition projection, which removes the register from the loop entirely.

IAMAllowedPrincipals

Lake Formation starts in "use only IAM access control" mode so existing Glue behaviour keeps working, and it does that by granting the IAMAllowedPrincipals group Super/All permissions on databases and tables. The consequence catches teams out: you can grant careful column-level permissions in Lake Formation and change nothing, because the table still carries that group grant and the catalog honours it. Moving genuinely to Lake Formation enforcement means registering the S3 location, revoking IAMAllowedPrincipals from the resource and from the database-creators list, and clearing the default catalog settings.

AWS Lake Formation Developer Guide — how hybrid access mode works

Hybrid access mode

The migration path that avoids a cutover. Register an S3 location in hybrid mode and Lake Formation enforces only CREATE_TABLE, CREATE_PARTITION and UPDATE_TABLE by default; you then opt specific principals in per database or table. An opted-in principal needs both Lake Formation permissions and IAM permissions; a principal who has not been opted in continues to be authorized by IAM alone. One subtlety decides questions: credential vending — Lake Formation handing back temporary, scoped S3 credentials — is unavailable unless the data location is actually registered, opted in or not.

AWS Lake Formation Developer Guide — hybrid access mode

Credential vending

The mechanism that makes table-level governance real. An integrated engine — Athena, AWS Glue, Amazon EMR, Redshift Spectrum — asks the Data Catalog for a table, the catalog checks the caller’s Lake Formation grants, and Lake Formation returns temporary credentials scoped to exactly the data the grant covers, along with any column, row or cell filters for the engine to apply. The engine reads S3 with those credentials, not with the user’s. That is why granting an analyst SELECT on three columns works without writing an S3 bucket policy, and why a directly attached s3:GetObject on the same prefix is a hole in the model rather than a convenience.

Data filter

Column-level filtering includes or excludes named columns; row-level filtering carries a predicate such as region = 'EU'; combining them gives cell-level access. The filter is applied by the engine on Lake Formation’s instruction, so the analyst runs an ordinary SELECT * and receives only the permitted cells, with no view to maintain and no second copy of the table. The usual exam shape is a single table that several teams must read with different visibility, where the wrong answers are "create a filtered copy per team" and "create a view per team" — both of which add objects that have to be kept in step.

LF-Tag

Tag-based access control inverts the maintenance burden. Instead of granting a team SELECT on each of 400 tables, you tag resources (classification=pii, domain=finance) and grant the team access to tag values. A new table tagged domain=finance is readable by the finance team the moment it is tagged, with no grant written — and tags inherit downward from database to table to column unless overridden. It is the standard answer whenever a stem combines "hundreds of tables", "dozens of teams" and "minimal ongoing administration".

Amazon DataZone

Where the Glue Data Catalog answers "what columns does this table have", DataZone answers "who owns this dataset, what does it mean, and how do I get approved to use it". Producers publish assets with business metadata into a domain’s inventory; consumers search, subscribe, and on approval get access provisioned through Lake Formation. The distinction that matters in a question: DataZone is about discovery and governed sharing between teams, and it does not replace the technical catalog underneath it. Its capabilities are also surfaced through the next generation of Amazon SageMaker, so either name may appear.

S3 storage class

Every class stores the same bytes with the same 11-nines durability design; what differs is the bargain. S3 Standard has no minimum duration and no retrieval charge. Standard-IA and One Zone-IA cost less per gigabyte, add a per-gigabyte retrieval charge and bill a minimum of 30 days. Glacier Instant Retrieval keeps milliseconds access at a 90-day minimum. Glacier Flexible Retrieval (90 days) and Glacier Deep Archive (180 days) require an explicit restore before the object can be read. One Zone-IA additionally stores in a single Availability Zone, so it is for data that can be regenerated. Delete or transition before the minimum and the remainder is billed anyway.

Amazon S3 User Guide — comparing the Amazon S3 storage classes

Minimum storage duration

The charge that turns a well-intentioned lifecycle policy into a bill. An object transitioned to Glacier Flexible Retrieval and deleted after 10 days is billed for the remaining 80, prorated — so a rule that tiers data aggressively and a retention rule that expires it soon after can cost more than leaving it in S3 Standard the whole time. Glacier Instant Retrieval and Glacier Flexible Retrieval are 90 days; Glacier Deep Archive is 180; Standard-IA and One Zone-IA are 30. S3 Standard and S3 Intelligent-Tiering have none.

Amazon S3 User Guide — transitioning objects using S3 Lifecycle

S3 Intelligent-Tiering

Objects start in Frequent Access and drop to Infrequent Access after 30 consecutive days without access, then to Archive Instant Access after 90 — all with milliseconds retrieval and no retrieval charge, and any access promotes the object straight back to Frequent Access. Two optional asynchronous tiers can be opted into: Archive Access from 90 days and Deep Archive Access from 180, which do require a restore. It is the right answer when the access pattern is unknown or changing, and the wrong one for a large population of very small objects: objects under 128 KB are not monitored and are always billed at Frequent Access rates, so the per-object monitoring fee buys nothing.

Restore request

Objects in Glacier Flexible Retrieval and Glacier Deep Archive cannot be read by GetObject at all; you call RestoreObject, wait, and then read a temporary copy that persists for the days you asked for. Retrieval tiers set the wait and the price: for Glacier Flexible Retrieval, Expedited 1–5 minutes, Standard 3–5 hours, Bulk 5–12 hours and free; for Deep Archive, Standard around 12 hours and Bulk within 48. Glacier Instant Retrieval needs no restore. A question that says "must be queryable on demand" has therefore already ruled out both restore-requiring classes, however cheap they are.

Amazon S3 FAQs — archive retrieval options

Transition waterfall

Standard → Standard-IA or One Zone-IA → Glacier Instant Retrieval → Glacier Flexible Retrieval → Glacier Deep Archive, in that direction only; Intelligent-Tiering sits beside the chain and manages its own tiers. Nothing in a lifecycle configuration can transition an object back to a warmer class — bringing it back means copying the object over itself with a new storage class, or a restore in the archive cases. Two further constraints bite: objects created after September 2024 and smaller than 128 KB will not transition to any class by default, and nothing can transition to Standard-IA or One Zone-IA before it is 30 days old.

Amazon S3 User Guide — constraints and considerations for transitions

Noncurrent version expiration

In a versioning-enabled bucket every overwrite keeps the old bytes as a noncurrent version and every delete writes a delete marker, so storage only ever increases. NoncurrentVersionExpiration takes a number of noncurrent days and, optionally, NewerNoncurrentVersions — keep the most recent N versions and expire the rest. Its partner rule, ExpiredObjectDeleteMarker, removes delete markers that no longer have any version behind them, which are otherwise invisible clutter that slows listings. A bill that keeps rising after a mass delete on a versioned bucket is this, every time.

Incomplete multipart upload

A large PutObject is really an initiate, many UploadPart calls and a complete; if the writer dies in the middle, the uploaded parts stay. They cost storage, do not appear in ListObjectsV2, and accumulate silently in buckets fed by failing jobs. The remedy is one lifecycle rule — AbortIncompleteMultipartUpload after, say, 7 days — which AWS recommends on effectively every bucket, and it is the answer whenever a stem describes storage charges that do not match the objects anyone can see.

S3 Storage Lens

The tool that tells you where the lifecycle problem is before you write a rule. A free default dashboard gives usage metrics for the whole account; advanced metrics add activity, prefix-level aggregation, retention of fifteen months of history and the recommendations. For a data engineer the useful views are the average object size per bucket — the small-file problem, quantified — the share of bytes in noncurrent versions, and the incomplete-multipart-upload bytes. It reports and advises; it changes nothing on its own.

S3 Inventory

Delivered daily or weekly as CSV, ORC or Parquet, carrying size, last-modified date, storage class, encryption status, replication status, and object-lock state. Because it is a file, you can register it in the Data Catalog and ask questions no API answers cheaply: how many objects are under 128 KB, which objects are unencrypted, which are still in Standard after a year. It also pairs with S3 Batch Operations, which takes an inventory report as its manifest — the standard way to retro-fit a change to billions of existing objects.

S3 Batch Operations

The answer whenever a change has to reach objects that already exist, because lifecycle rules act on age and class rather than on an arbitrary list. The manifest is an S3 Inventory report or a CSV you supply; the job retries, tracks per-object success and failure, and writes a completion report. The two tasks it is usually the answer to: restoring a large archived set ahead of an audit (initiating RestoreObject on millions of keys), and re-encrypting or re-tagging historical data to meet a policy written after the data landed.

DynamoDB TTL

Enable TTL on the table, name the attribute, and store a Unix epoch timestamp in seconds as a Number; an item whose attribute is a different type is ignored entirely, which is the commonest silent failure. AWS documents deletion as "typically within a few days" of expiry, so expired items remain readable in the meantime and the application must filter them out of Query and Scan results itself. Deletes appear in DynamoDB Streams as service deletions rather than user deletes and remove the item from its indexes. With global tables, the delete in the originating Region consumes no write capacity but each replicated delete does.

Amazon DynamoDB Developer Guide — using time to live (TTL)

Point-in-time recovery (PITR)

In DynamoDB, PITR is a switch rather than a schedule: once on, any second in the window is restorable, which is the control for the failure mode backups handle badly — a bad deployment that corrupts rows for forty minutes before anyone notices. Two properties shape answers. The restore always creates a new table, so recovery is a cutover and not an in-place rewind. And it is the complement of on-demand backups rather than a replacement: PITR covers recent operational accidents, while on-demand backups (and AWS Backup) cover long retention for compliance.

Redshift snapshot

Automated snapshots are taken on a schedule and expire after their retention period (one day by default, configurable up to 35); manual snapshots have no expiry unless you set one, so they are the mechanism for keeping a quarter-end state for years. Either can be copied to a second Region for disaster recovery, and either can be restored into a new cluster — including a table-level restore from a snapshot into an existing cluster, which is the surgical answer when one table was truncated and the rest of the warehouse is fine.

RPO and RTO (Recovery point objective / recovery time objective)

They decide the mechanism before any product is named. An RPO of zero rules out any scheduled backup and demands synchronous replication or a replayable source; an RPO of 24 hours makes a nightly snapshot sufficient. An RTO of minutes rules out restoring hundreds of terabytes from Glacier Deep Archive, whose retrieval alone is around 12 hours. The common design error is treating cheap archival storage and a short RTO as compatible: the retrieval time is part of the recovery time, and the cheapest class is the slowest to come back.

Durability vs availability

S3 Standard, Standard-IA and the Glacier classes are all designed for the same 99.999999999% durability, because they all replicate across at least three Availability Zones — what varies is the availability SLA and the time to first byte. One Zone-IA is the exception that proves the rule: it is the only class that lowers durability expectations, because a single Availability Zone’s loss takes the data with it. And neither property protects against a deletion you authorised, which is why versioning, object lock and backups answer a different question from "how durable is the class".

S3 Replication

Requires versioning on both sides and a replication configuration naming a role. It copies new objects, and existing objects only if you run batch replication. Three variants answer three different stems: cross-Region replication for a disaster-recovery copy or to put data near a consumer, same-Region replication for aggregating logs into one account or changing ownership, and Replication Time Control when a contractual replication window has to be measured — it replicates most objects in seconds and 99.99% within 15 minutes, backed by a service-level agreement, and emits replication metrics and events so a breach is visible. Replication copies deletes only if you ask, and never replicates a delete of a specific version.

Star schema

The dimensional model’s default shape, and on a columnar MPP warehouse it is a performance decision rather than a stylistic one. Each dimension is a single flat table holding the whole hierarchy (product_key, product_name, subcategory, category, department), so the planner joins the fact to each dimension exactly once and never chases a chain. On Redshift the physical mapping follows: the fact table and its largest dimension share a distribution key so that join is collocated, the small dimensions use ALL distribution so they are present on every node, and the fact’s sort key is the date column every query filters on.

Snowflake schema

Instead of one dim_product holding category and department as columns, a snowflake has dim_product → dim_subcategory → dim_category. The saving is real where a dimension is genuinely enormous or where an attribute is maintained by a separate owner; the cost is that any query grouping by department now joins three tables instead of one, and on a distributed warehouse at most one of those joins can be collocated. Because columnar compression already removes most of the redundancy that normalisation was invented to avoid, the storage argument is far weaker here than in an OLTP schema — which is why the warehouse default is the flat star, and snowflaking is a justified exception.

Fact table

A fact row records something that happened — a line on an order, a sensor reading, a page view — and holds the additive numbers plus the keys that describe the context. It grows without bound and is never updated in place in a well-built warehouse, which is why it is the table whose sort key and distribution key matter most. Three varieties recur: a transaction fact (one row per event), a periodic snapshot (one row per entity per period, for balances that only make sense at a point in time), and an accumulating snapshot (one row per process instance, updated as it passes milestones) — the one genuine exception to append-only.

Dimension table

Dimensions supply the words in "revenue by region for enterprise customers". They are small enough to denormalise freely and to replicate to every node, and they are where attribute change has to be handled, because a customer’s region can change while the facts already recorded against it must not. That is the slowly-changing-dimension problem. A conformed dimension is one shared verbatim by several fact tables, which is what allows two subject areas to be compared at all — the same dim_date and dim_customer across sales and support is the difference between a warehouse and two reports.

Grain

"One row per order line per shipment" is a grain; "sales data" is not. Everything downstream depends on it: which dimensions can attach (only those that are single-valued at the grain), whether a measure is additive, and whether a join will fan out and inflate totals. The classic failure is mixing grains in one table — order-header discounts stored beside order-line amounts — after which summing the discount column multiplies it by the number of lines, and no amount of SQL care fixes the model. Finer grain costs storage and buys answerable questions; coarser grain forecloses them permanently.

Surrogate key

Two reasons it exists, and the second is the one that gets tested. It decouples the warehouse from the source, so a reissued or reformatted business key, or a merger that brings two customer_id spaces together, does not break existing facts. And it is what makes Type 2 history possible at all: the same customer must be able to have several dimension rows — one per version — and the natural key cannot identify them because it is the same on every one. Facts therefore join on the surrogate key, which pins each fact to the version of the dimension that was current when the fact occurred.

Denormalisation

Normalisation was designed to stop update anomalies in systems that update single rows constantly. An analytical store rarely updates anything and reads enormous ranges, so the trade reverses: a repeated category name compresses to almost nothing in a columnar encoding, while the join it removes would have cost real work on every query. In DynamoDB it goes further and becomes mandatory — there are no joins, so anything a request needs together must be stored together. The discipline that keeps it honest is a single writer: the duplicate must be produced by one pipeline from one source, never maintained in two places by hand.

Single-table design

DynamoDB has no joins, so the only way to fetch related entities together is to have stored them together under one partition key. A single-table design names its keys PK and SK rather than CustomerId and OrderId, and encodes the entity type into the value: PK=CUST#42 with sort keys PROFILE, ADDR#1, ORDER#2026-03-14#9931. One Query on PK=CUST#42 then returns the customer and their orders in a single call. The method is the reverse of relational design: enumerate the access patterns first, then derive keys that satisfy them. Its costs are real — the table is hard to read by eye, and a genuinely new access pattern may need a new index or a backfill.

Amazon DynamoDB Developer Guide — data modeling building blocks

Item collection

The unit of co-location, and therefore the unit of design. Because a collection is contiguous and sorted, Query can return a prefix range of it cheaply — "the customer record plus their last ten orders" is one request reading one contiguous run of items. Two limits shape the model: a collection that attracts disproportionate traffic is a hot partition, bounded by the per-partition throughput ceiling; and a collection with a local secondary index is capped at 10 GB, which is the main practical reason to prefer a global secondary index.

Composite sort key

A sort key of ORDER#2026#03#14#9931 can be queried with begins_with(SK, 'ORDER#2026#03') to get a month, ORDER#2026' to get a year, or compared with between to get a range — one stored attribute serving several hierarchy levels. Two rules make it work: the components must run coarse to fine (year before month before day), because a prefix match can only cut from the left; and anything compared by range must be lexicographically ordered, which is why dates are stored zero-padded as 2026-03-14 and not 14/3/2026, and why numbers used in a sort key are padded to a fixed width.

Index overloading

A table is limited in how many indexes it can carry, so a single-table design gives its index keys generic names — GSI1PK, GSI1SK — and lets each entity type populate them with whatever that entity needs to be found by. An order might set GSI1PK=STATUS#OPEN to support "all open orders", while an employee row sets GSI1PK=DEPT#FINANCE for "everyone in finance"; one index serves both because queries always supply the prefix. It works because a global secondary index is sparse: an item that does not set the index’s key attributes is simply absent from it.

Sparse index

Write an unresolved attribute only on tickets that are open, index on it, and the index holds exactly the open tickets: scanning it is cheap no matter how many closed tickets exist, and closing a ticket removes it from the index by deleting the attribute. That is the idiomatic DynamoDB answer to "find all items in state X" when X is rare. The same mechanism is also a troubleshooting answer — items "missing" from a global secondary index are usually items that never had the index key, not a replication delay, though genuine propagation does take about a second and an under-provisioned index can throttle the base table’s writes.

Amazon DynamoDB Developer Guide — global secondary indexes

Write sharding

The fix when a legitimate key is too popular — a leaderboard’s current day, a single device writing at high rate. Append a calculated or random suffix (DAY#2026-03-14#7) so writes spread over N partitions, each with its own throughput allowance. The price is on the read side: retrieving the whole logical collection now means N queries fanned out and merged, so N should be the smallest number that clears the ceiling. It is specifically the answer to per-partition throttling, which adaptive capacity cannot solve for a single key value because one key’s items cannot be split further.

Amazon DynamoDB Developer Guide — using write sharding to distribute workloads evenly

Apache Iceberg

Three layers. The catalog (on AWS, usually the Glue Data Catalog) holds a pointer to the current metadata file. The metadata layer holds the metadata file — schema, partition spec, properties, and the list of snapshots — plus manifest lists and manifest files that record every data file with its partition values and per-column minimum/maximum statistics. The data layer is ordinary Parquet, ORC or Avro. A write adds files and then atomically swaps the metadata pointer, so readers see either the old table or the new one and never a half-written state. Because files are listed rather than discovered, the engine prunes on statistics instead of on directory names, and partitioning can change without rewriting data.

AWS Prescriptive Guidance — Apache Iceberg on AWS

Snapshot (table format)

Every commit creates a new snapshot and lists it in the table metadata; a reader resolves the current snapshot when it loads the table and then sees a consistent set of files for the whole query, unaffected by concurrent writers until it refreshes. Snapshots share manifest files where nothing changed, so the metadata does not duplicate per commit. They are also the thing that has to be cleaned up: expiring snapshots is what finally allows the superseded data files to be deleted, and it is what shortens your time-travel window. On Amazon S3 Tables the defaults keep at least 1 snapshot and expire at 120 hours.

Time travel

In Athena against an Iceberg table this is FOR TIMESTAMP AS OF or FOR VERSION AS OF, and it costs nothing to keep because the older files already exist until snapshots are expired. It answers three real questions: reproducing a report exactly as it ran last Tuesday, diffing before and after a suspect load, and rolling a table back after a bad merge without restoring a backup. The limit is the retention policy — once a snapshot is expired and its unreferenced files removed, that point in time is gone, so the snapshot-retention setting is the time-travel window and should be chosen as such.

Copy-on-write (CoW)

Changing one row in a 256 MB Parquet file means writing a new 256 MB file with that row altered and pointing the metadata at it. Write amplification is therefore high and proportional to the files touched, while a reader opens exactly one set of columnar files and merges nothing. It is the default in Apache Hudi and the right mode for read-heavy tables whose data changes rarely — a dimension refreshed nightly, a curated mart. It is the wrong mode for a continuous stream of small updates, where every few seconds a large file is rewritten for a handful of rows.

Amazon EMR Release Guide — how Hudi works

Merge-on-read (MoR)

Hudi logs changes to row-based Avro delta files beside the columnar base files; Iceberg encodes them as delete files. Either way the write is small and fast, and the cost moves to the reader, which must apply the deltas on the fly — a read-optimised query can instead skip them and return the last compacted state, trading freshness for speed. Compaction is mandatory, not optional: without it the delta count grows until reads crawl. The choice is a sentence long — write-heavy or streaming updates with tolerant readers take merge-on-read; read-heavy tables with occasional updates take copy-on-write — and in Hudi the table type is fixed at creation.

Amazon EMR Release Guide — how Hudi works

Apache Hudi

Hudi’s distinguishing feature is its index: a mapping from record key to the file holding that key, which lets an upsert find and rewrite only the affected files instead of joining the incoming batch against the whole table. Its second is the incremental query — give it two commit times and it returns only the records that changed between them, which makes a downstream job genuinely incremental rather than a full recompute with a filter. Table type (copy-on-write or merge-on-read) is chosen at creation and cannot be changed. Supported natively by AWS Glue, Amazon EMR, Athena and Redshift.

Delta Lake

A _delta_log directory holds one JSON file per commit describing the files added and removed, with Parquet checkpoints every so often so a reader does not replay the whole history. The result is the same guarantee set as Iceberg and Hudi — atomic commits, snapshot isolation, time travel, MERGE — reached by a different route. On AWS it is supported by AWS Glue, Amazon EMR, Athena and Redshift, and the honest reason to pick it is usually organisational: an existing Databricks or Spark estate already writes Delta. For a greenfield AWS lake the ecosystem gravity is with Iceberg, which is what Amazon S3 Tables manages.

Partition evolution

A Hive-style table’s partitioning is its directory layout, so repartitioning from daily to hourly means rewriting every file into new prefixes and re-registering every partition. Iceberg stores each data file’s partition values in its manifest, so the table can simply start writing with a new partition spec: old files keep their old spec, new files use the new one, and the planner prunes both correctly in one query. The related trick is hidden partitioning — the table derives the partition value from a column with a transform such as days(event_time), so a query filtering on event_time prunes without anyone writing a separate partition predicate.

Compaction

Streaming writers and merge-on-read tables both produce the same problem — a growing population of small files and delta or delete files that every read has to reconcile. Compaction merges them into large, sorted base files; strategies include simple bin-packing by size, sorting on the table’s sort order, and z-ordering for multi-column locality. Run it too rarely and queries degrade; run it too often and you pay to rewrite data that will be rewritten again. Amazon S3 Tables manages it, targeting 512 MB files by default (configurable 64–512 MiB), and self-managed Iceberg tables need an OPTIMIZE or equivalent job on a schedule.

MERGE INTO

The statement a table format makes possible on a data lake: MERGE INTO target USING source ON keys WHEN MATCHED THEN UPDATE … WHEN NOT MATCHED THEN INSERT …, executed as one atomic commit. It is how change data capture lands, and it is how a slowly-changing dimension is maintained in a single pass — expire the matched current row and insert the new version in the same statement. Two cautions: a source containing two rows for the same key is ambiguous and engines reject it, so the source must be deduplicated to the latest change per key first; and on a copy-on-write table each merge rewrites every file it touches.

Schema evolution

Four changes, in rising order of danger. Adding an optional column is almost always safe — old readers ignore it, new readers see null for old data. Removing a column breaks any reader that still selects it. Renaming is a remove plus an add unless the format resolves by name and the engine is told about the rename. Changing a type is safe only in the widening direction (int → bigint, float → double); narrowing loses data and reinterpreting it (string → date) fails on any value that does not parse. The engineering question is never "can I change it" but "who reads this, and which of those four is this".

AWS Glue Schema Registry

Moves schema enforcement from read time to write time. A producer serialises with a schema id instead of the whole schema, so messages get smaller, and the registry validates each new version against the compatibility mode before accepting it — a producer that drops a required field under BACKWARD fails at registration rather than breaking consumers hours later. It integrates with Apache Kafka and Amazon MSK, Amazon Kinesis Data Streams, Apache Flink and AWS Lambda. Its limit is scope: it governs the streams it is wired into, not files that some other job drops into a bucket.

AWS Glue Developer Guide — Schema Registry

Compatibility mode

Decide it from who upgrades first. BACKWARD — the recommended default — lets a consumer on the new schema read data written with the previous one, so consumers can be upgraded before producers; it therefore forbids dropping a field or changing its type. FORWARD lets a consumer on the old schema read data written with the next one, so producers may move first; it permits adding fields and deleting optional ones. FULL requires both. The _ALL variants check against every previous version rather than only the last, which is what you need when old consumers may be arbitrarily far behind. NONE accepts anything without checking; DISABLED forbids new versions altogether.

AWS Glue API Reference — CreateSchema compatibility

Column resolution (by name or by index)

Athena reads Parquet by name (parquet.column.index.access defaults to false) and ORC by index (orc.column.index.access defaults to true). The consequences are exactly opposite. In Parquet, reordering or inserting a column is harmless and renaming one makes its data vanish into nulls. In ORC, renaming is harmless and reordering silently serves the wrong column’s values under the right heading — the worst kind of failure, because nothing errors. Either default can be overridden with a CREATE TABLE carrying the SerDe property, but the property is not propagated to existing partitions, which have to be re-added.

Amazon Athena User Guide — handling schema updates

Slowly changing dimension (SCD)

When a customer moves from the West region to the East, the warehouse has to decide what last year’s sales were. Type 0 never changes the value — correct for things that genuinely cannot change, like a date of birth. Type 1 overwrites, so history is restated as if the customer had always been in the East. Type 2 closes the current row and inserts a new one, so each fact stays joined to the attributes that were true when it happened. Type 3 adds a previous_region column, which answers one before-and-after comparison and nothing deeper. The question that picks between them is never technical: it is whether the business reports history as it was or as it is now.

Type 2 dimension

A Type 2 row carries the natural key, a new surrogate key, effective_from, effective_to (open rows usually 9999-12-31 rather than null, so range predicates work) and often is_current. The load closes the old row and inserts the new one, typically as one MERGE. The costs are the part people skip. Every query must now qualify which version it wants — a lookup by natural key returns several rows, so either is_current = true or a date-range join is mandatory, and omitting it quietly multiplies totals. The dimension grows with change rather than with population. And the ETL must detect change by comparing attributes, which means reading the current rows on every load.

AWS Glue trigger

Glue’s own starter motor, and the reason a simple two-job dependency needs no external orchestrator. Three kinds: on-demand (you call it), scheduled (a cron expression), and conditional — fire when a watched set of jobs or crawlers reaches a given run state, with a logical operator of ANY or ALL across the watch list. Conditional triggers chained together are a Glue workflow. The state you watch matters: watching for SUCCEEDED builds a happy-path chain, and watching for FAILED is how a cleanup or alerting branch is wired. Where this runs out is anything with a loop, a map over a list, or a human approval — at which point you want Step Functions or MWAA.

transformation_ctx

A bookmark is not one number for the job; it is a map from operator to state, and transformation_ctx is the key. Pass it on every source read and on the write, keep it unique per operator, and never change it casually — renaming it orphans the old state, so the next run looks up a key that does not exist and reads the source from the beginning. Omitting it has the mirror effect: that operator is simply not tracked and reads everything every run, which is occasionally what you want (a small dimension table that must be re-read in full beside an incremental fact feed) and is otherwise the explanation for "the bookmark is on but it reprocesses".

Bookmark key

For a JDBC source there are no object timestamps to compare, so Glue needs a column whose values only ever increase: a sequence id, an insert timestamp, a monotonic version. It uses the table’s primary key by default, and several columns combine into one compound key. The soundness condition is the whole lesson — if the key can be reused, back-filled or issued out of order, rows below the stored high value will never be read again, and the loss is invisible because the job succeeds. A natural key such as an order number assigned by an upstream system that occasionally replays a block of ids is exactly the trap; a monotonically increasing updated_at written by the database is exactly the fix. Updates to existing rows are the other gap: a bookmark on an insert-time column never sees a row that was changed, which is where change data capture belongs instead.

Idempotent pipeline step

Everything in a data pipeline is retried eventually: a trigger fires twice, a notification is delivered twice, an operator reruns a failed task, an engineer reruns yesterday. Idempotency is what makes that harmless, and it is a design property rather than a service feature. The reliable implementations are few: overwrite a partition rather than appending to it (INSERT OVERWRITE on one date rather than INSERT INTO), write to a deterministic key so the second write replaces the first, MERGE on a business key into an open table format, or carry a deduplication id the sink enforces. The unreliable implementation is "append, and hope it only runs once". Test for it directly: run the step twice on the same input and compare row counts.

Lambda event source mapping

With a stream or queue source you do not call Lambda; a mapping polls on your behalf, and its settings decide the pipeline’s behaviour far more than your code does. BatchSize and MaximumBatchingWindowInSeconds trade latency for efficiency. ParallelizationFactor runs up to ten concurrent invocations per shard while preserving per-partition-key order. For stream sources the mapping checkpoints per shard, so a batch that keeps failing blocks that shard until it expires — the poison-pill problem — which is why BisectBatchOnFunctionError, MaximumRetryAttempts, MaximumRecordAgeInSeconds and an on-failure DestinationConfig exist. Report partial batch success (ReportBatchItemFailures) and only the failed records are retried rather than the whole batch. None of this applies to a synchronous invoke, which is why "Lambda is slow" and "the mapping is misconfigured" are different diagnoses.

Reserved concurrency

Two effects from one setting, and the second is the one that solves data-pipeline problems. It guarantees the function that many simultaneous executions, carved out of the account’s regional pool so no noisy neighbour can starve it; and it limits the function to that many, which is the standard way to protect a fragile downstream — a relational database with two hundred connections does not want two thousand concurrent writers, so you reserve a number the database can survive and let the queue absorb the rest. Setting it to zero throttles the function completely, which is the documented way to stop a runaway consumer without deleting its trigger. Distinguish it from provisioned concurrency, which pre-initialises execution environments to remove cold starts and is billed for whether used or not: reserved concurrency shapes how many, provisioned concurrency shapes how fast the first one starts.

Distributed Map

The inline Map state runs a modest fan-out inside one execution and is bounded by that execution’s history limit. Distributed Map instead starts child executions — up to 10,000 concurrent — and can take its items straight from an S3 object listing, a JSON array in S3, or the rows of a CSV, which makes "process every file in this prefix" a workflow rather than a script. It carries a tolerated failure percentage and count, so a batch of fifty thousand files can be allowed to lose a few without failing the whole run, and it writes per-item results back to S3. For a data engineer this is the usual answer to a per-partition or per-file backfill that must be both parallel and resumable, and it is why backfill shape, not backfill code, is what you design.

MaxConcurrentRuns

A Glue job’s own concurrency ceiling, set in its execution property. The default is 1, and a StartJobRun beyond the ceiling fails with ConcurrentRunsExceededException unless job queuing is enabled. Two reasons to think about it rather than raising it reflexively: a job with bookmarks enabled and concurrency above one can have two runs reading and advancing the same state, which is one of the documented causes of reprocessed data; and the per-job limit sits underneath two account-level limits — maximum concurrent job runs per account and maximum task DPUs per account — so raising the job setting does not raise the ceiling you will actually hit at scale.

AWS Glue job queuing

Without it, hitting a concurrency limit or the account DPU limit returns an error and the caller owns the retry — which is how an event-driven pipeline with a morning burst ends up with a hand-written retry loop and a dead-letter queue full of ConcurrentRunsExceededException. With job queuing enabled, Glue enqueues the run and reattempts it as resources free, so burstiness becomes latency rather than failure. The trade is exactly that: a queued run is a run that has not started, so an alarm on job duration measured from submission will now fire for reasons that are not the job’s fault, and a genuinely time-critical job still needs capacity rather than a queue.

Throttling ceiling

Every service in a data pipeline refuses work at some point, and the refusal is normal operation rather than a fault. The ones that decide real designs: Athena’s active DML query quota, which counts running and queued queries and answers TooManyRequestsException beyond it; Glue’s per-job and per-account concurrency plus the account task-DPU limit; Lambda’s regional concurrency pool; Kinesis Data Streams at 1 MB/s or 1,000 records/s per shard for writes and 2 MB/s per shard for reads; DynamoDB per-partition limits; and API rate limits on the control planes everything calls. The design response is always one of four: back off with jitter, queue, reserve dedicated capacity (Athena capacity reservations, provisioned throughput), or request a quota increase — and knowing which of the four a scenario is asking for is most of the question.

Column pruning

In a columnar file the values of one column sit together in their own compressed chunk, so an engine asked for three of ninety columns reads three chunks and ignores the rest. In a row-oriented file — CSV, JSON, a text log — the columns of one record are interleaved, so there is no way to reach the third field without reading the first two and the rest of the line; the whole record is read regardless of what the SELECT list says. This is why "replace SELECT * with named columns" is excellent advice on Parquet and no help at all on gzipped JSON, and why a question that offers column pruning as the fix for an un-converted raw table is offering a distractor. Parquet’s per-chunk min/max statistics add a second effect on top: chunks that cannot satisfy a filter are skipped without decompression.

Athena workgroup

Workgroups separate teams and workloads inside one account. Each carries its own query result location and encryption settings, its own engine version, its own CloudWatch metrics (publishable per workgroup), its own tags for cost allocation, and its own data usage controls. Access is governed by IAM on the workgroup resource, so "the marketing team may run queries but only in their workgroup, with a per-query scan cap" is a workgroup answer rather than a policy-on-tables answer. Two settings worth knowing because they decide behaviour for everybody in the group: you can override client-side settings so no analyst can escape the configured result location, and you can enable result reuse and capacity at the workgroup level.

Athena data usage control

Two distinct controls that answer two distinct questions. The per-query limit is a hard ceiling on one query’s scan — exceed it and Athena cancels the query, which is how you stop a single SELECT * over the whole lake before it bills. The per-workgroup control watches total scanned bytes over a time period and fires an alarm action (notify, or invoke something) when it is crossed, without cancelling anything. Note what neither does: they do not make any query cheaper or faster. They cap exposure and they create a signal; the actual cost reduction still comes from partitioning, projection and columnar storage. A question that asks for lower cost without losing capability is not answered by a usage limit, because a cancelled query is lost capability.

Athena query result reuse

When enabled, Athena checks whether this exact query has already been answered recently enough and, if so, returns the stored result without scanning anything — so the query costs nothing and returns immediately. The age is yours to choose: MaxAgeInMinutes defaults to 60 and the maximum is 10,080 minutes, which is 7 days. That knob is the whole design decision, because reuse trades freshness for cost by exactly the amount you set. It is the right answer for a dashboard that refreshes every five minutes against data that lands hourly, and the wrong answer for an operational query whose entire point is to see the newest rows. Distinguish it from Redshift result caching, which is on by default and invalidates on base-table change: Athena’s reuse is explicit, time-bounded, and does not notice that new objects landed in S3.

Athena capacity reservation

The other Athena billing model. Instead of paying for bytes scanned per query, you reserve processing capacity measured in DPUs and pay by the hour, assigning selected workgroups to the reservation. Two requirements point here and nothing else does: predictable spend independent of how much data analysts happen to touch, and guaranteed concurrency — reserved capacity is not competing for the account’s shared active-query quota, so a critical workload stops being queued behind an exploratory one. Both billing models can coexist in one account, workgroup by workgroup. The trade is utilisation: an idle reservation still bills, so it pays off for a steady workload and loses to per-query billing for a bursty one.

CTAS (CREATE TABLE AS SELECT)

The transformation primitive you get without leaving SQL. CREATE TABLE … WITH (format = 'PARQUET', partitioned_by = ARRAY['dt'], bucketed_by = …, write_compression = 'SNAPPY') AS SELECT … reads a raw table once and writes a converted, partitioned, compressed copy, registering it in the catalog. That makes CTAS the cheapest honest answer to "our table is gzipped JSON and queries cost too much": pay once to convert, then every subsequent query reads Parquet. INSERT INTO … SELECT is its incremental sibling for appending a new day, and both are how the small-files problem gets fixed, since the writer’s output file count is controllable where a streaming producer’s is not. One limit to remember: CTAS has a bucketing-and-partition count ceiling, so a conversion partitioned too finely fails rather than running slowly.

Redshift federated query

Where Spectrum reaches into S3, federated query reaches into an operational database: Amazon RDS for PostgreSQL, Aurora PostgreSQL, RDS for MySQL and Aurora MySQL. The leader node connects to the remote instance for metadata; compute nodes then issue subqueries with the predicate pushed down and distribute the result rows for further processing. Credentials come from Secrets Manager, and remote activity is visible in SVL_FEDERATED_QUERY. Its proper use is small, fresh reference data joined to large warehouse history — a current price list, today’s open orders — not bulk extraction, because every row crosses a database connection. When a scenario says live or without building an ETL pipeline about an RDS table, this is the feature; when it says the data lake, it is Spectrum.

Concurrency scaling

The answer to queries queue at 9 a.m. rather than queries are slow all day. When queuing is detected, Redshift routes eligible queries to additional transient clusters, so throughput rises without resizing anything, and it is enabled per workload management queue rather than globally. It accrues one hour of free credit per day of main-cluster uptime, which covers the bursts of most clusters outright. What it does not fix is a single slow query — a badly distributed join is just as slow on a transient cluster — and that distinction is the usual question: concurrency problems scale out, per-query problems need distribution keys, sort keys or a materialized view.

Redshift workload management

WLM decides which query runs where. Automatic WLM — the current default — assigns memory and concurrency dynamically and takes a query priority per queue, which is the lever you actually want: ETL loads at high priority overnight, ad-hoc exploration at low priority during the day. Queues route by user group or query group label. On top of that, query monitoring rules watch metrics for a running query (execution time, rows scanned, nested-loop joins, spill to disk) and take an action — log, hop to another queue, or abort. That is how one analyst’s runaway cross join stops being everybody’s incident, and it is the control to name when a scenario asks for a guardrail rather than more capacity.

Redshift result caching

Transparent and free: when a query arrives, the leader node looks for a valid cached result and, if it finds one, returns it without running anything. The conditions are the exam material. The user must have permission on the objects; the tables or views must not have been modified; the query must not call a function that has to be evaluated every time, such as GETDATE; the query must not reference Spectrum external tables; relevant configuration parameters must be unchanged; and the query text must match syntactically. Session-level control is enable_result_cache_for_session, and SVL_QLOG.source_query tells you after the fact whether a cache hit occurred. Contrast with Athena result reuse, which is opt-in, age-bounded, and does not invalidate when new data lands.

SPICE (Super-fast, Parallel, In-memory Calculation Engine)

The choice that decides whether a BI dashboard is fast or fresh. Direct query sends every visual to the underlying source, so the data is current and every refresh costs an Athena scan or a Redshift query. SPICE imports a snapshot and serves visuals from memory, so dashboards are fast and independent of the warehouse, at the cost of being exactly as current as the last refresh. Refreshes come as full (replace everything) or incremental — SQL-based sources only, within a look-back window on a date column, deleting and replacing that window — with up to five schedules per dataset and an API for refreshes in between. Two operational facts that make SPICE a staleness source worth monitoring: a failed refresh leaves users looking at the previous data with no visible error on the dashboard, and the dataset stays available throughout a refresh, so nothing about the UI reveals the age of what you are reading.

Index State Management (OpenSearch Index State Management)

The lifecycle engine inside an OpenSearch Service domain, so log retention needs no external cron. A policy defines states and the actions and transitions between them: roll over a daily index, reduce replicas, snapshot, migrate to UltraWarm, migrate to cold, delete. Two mechanical facts cause most of the real incidents. The tiers must be traversed in order — hot → UltraWarm → cold on the way out, and the reverse on the way back — and the policy must contain the explicit migration action (warm_migration, cold_migration); a state merely named "warm" changes the state label and leaves the index sitting in hot storage, burning hot-tier cost while the dashboard says the policy is working. Check with the UltraWarm migration status API rather than by reading the policy.

UltraWarm

The middle tier of the three. Hot nodes hold attached storage for active indexing and the lowest-latency queries; UltraWarm nodes keep index data in S3 with a local cache, serving read-only indexes at a fraction of hot-tier cost and scaling to petabytes; cold storage detaches the data from compute entirely for unlimited archival, and an index there must be migrated back to UltraWarm before it can be searched. The decision is one sentence long: an index you still write to must stay hot, an index you still search belongs in UltraWarm, and an index you keep only because someone might ask belongs in cold — or in S3 behind Athena, which is cheaper still if you can accept SQL-over-files instead of search.

Treat missing data

Each data point in an alarm’s evaluation window is not breaching, breaching, or missing, and this setting says how to score the missing ones: notBreaching treats them as good, breaching treats them as bad, ignore holds the current state, and missing — the default — sends the alarm to INSUFFICIENT_DATA when the whole window is empty. That default is the mechanical reason a "failed runs > 0" alarm is useless for detecting a pipeline that stopped: a job that never started publishes no failure metric, the window is empty, and the alarm sits in INSUFFICIENT_DATA for ever. Two correct fixes: alarm on a heartbeat metric with breaching so absence is the signal, or alarm on a freshness measurement that is always published. The general rule — breaching for anything that should report continuously, notBreaching for a metric that only exists when something goes wrong.

Freshness check

Everything else in a pipeline monitors activity; freshness monitors result. You compute the age of the latest partition, the maximum event timestamp, or the newest object’s last-modified time, publish it as a custom CloudWatch metric on a schedule, and alarm when it exceeds the promise you made to consumers. Because the measurement is emitted by your own scheduled check rather than by the pipeline, it keeps reporting when the pipeline is doing nothing at all — which is exactly the case a failure alarm cannot see, for the reason described under treat missing data. Glue Data Quality expresses the same idea declaratively with its DataFreshness rule, in hours or days against a date column. State it as a service level: the orders table is never more than 90 minutes behind. That sentence is both the alarm threshold and the thing you owe your consumers.

Composite alarm

A rule like ALARM(job-failed) AND NOT ALARM(upstream-outage) over existing alarms, with its own actions. Two jobs for it. Noise suppression: when one upstream outage trips thirty downstream alarms, the composite is what pages, and the thirty stay as diagnostic detail rather than thirty messages. Expressing a real condition: "the pipeline is unhealthy" genuinely is a conjunction — the job failed and the data is stale and the retry queue is filling — and a composite says that in one place. It can also suppress child alarms while a known parent condition is active, which is the documented way to stop maintenance windows generating pages. It adds no new measurement; if none of your alarms can see the problem, a composite of them still cannot.

CloudWatch anomaly detection

For a metric with a shape rather than a limit. CloudWatch trains on up to two weeks of history, builds an expected band, and lets an alarm fire when values leave it — so "rows ingested" can be alarmed without anybody deciding what number is normal for a Sunday. This is the right tool for the data-volume signals a fixed threshold handles badly: a Monday that is five times a Sunday is not an incident, and a Monday that is a tenth of last Monday is, even though both are inside any static range wide enough to hold both. The cost is a warm-up period and a model that will learn a bad pattern if the bad pattern persists, which is why it complements rather than replaces a freshness alarm with an explicit promise behind it.

Metric math

Services publish counters; operations cares about ratios. Metric math closes that gap inside CloudWatch: divide failed runs by total runs to alarm on a failure rate rather than a count, use FILL to turn a sparse metric into a continuous one, SEARCH to aggregate across every job in an account without naming them, and RATE/DIFF to turn a cumulative counter into a per-interval change. For a data engineer the common uses are a rows-in versus rows-out ratio across a transformation, a per-job share of a shared DPU quota, and a derived lag in seconds from two timestamps. It is also the cheap way to convert a metric that only exists on error into one an alarm can evaluate continuously.

Glue observability metrics

The telemetry that answers why a Glue job is slow rather than whether it failed. Published every 30 seconds in four categories, and two metrics carry most of the diagnostic weight. glue.driver.skewness.stage compares the longest task in a stage to the median; it reads 0 while that ratio stays under a stage skewness factor (default 5, overridable through a Spark configuration property) and rises from there, with glue.driver.skewness.job taking the duration-weighted maximum across stages so a brief skewed stage is not mistaken for a job-wide problem. glue.driver.workerUtilization says how much of the capacity you paid for was used — a reading near 0.2 is the signal to enable auto scaling or shrink the job rather than to tune the code. Alongside them sit reliability metrics that group failures by reason and throughput metrics for bytes and records read and written.

Glue job run insights

Glue’s own first-pass diagnosis, on by default. When a run fails, you may get an exception analysis stream named <job-run-id>-job-insights-rca-driver carrying the line number of your script that failed, the last Spark action in the query plan, and a concise time-ordered sequence of driver and executor events with the failed task and executor identified — and a separate rule-based insights stream carrying a root cause and a recommended action, such as a specific job parameter to set. Two facts worth knowing before a scenario asks: neither stream is created when a run succeeds, so this is useless for diagnosing a job that completes with wrong output, and the second stream exists only when a service-defined rule matches, so its absence is not evidence of anything.

Glue continuous logging

Without it, a Glue job’s logs become useful at the moment the job stops being interesting. Continuous logging pushes driver and executor output to CloudWatch during the run, with a filtered -driver stream that strips the Spark framework noise so your own print and logger lines are readable, and a progress bar reporting stage completion. For a long job this is the difference between watching a five-hour run and waiting for it. The practical pairing: continuous logging for what the job is doing now, job run insights for why it failed, the Spark UI for which stage and which task, and observability metrics for whether this is normal for this job.

Spark UI and history server

The only surface that tells you which stage of a Spark job is the problem. Glue and EMR can write Spark event logs to S3, and the history server renders them: a timeline of stages, task duration distributions, input and shuffle read/write bytes, spill to disk, and the executor breakdown. Three readings do most of the work. A stage whose task-duration distribution has a long right tail is skewed. Large shuffle write with small input means a join or wide aggregation is moving everything across the network. Spill to disk means the executor memory is too small for the partition size — which is a partition-count or worker-type decision, not a code one. The event logs persist after the cluster or job is gone, which is what makes a post-mortem possible at all.

Consumer lag

The same idea as iterator age in Kafka vocabulary. A topic partition has a latest offset; a consumer group has a committed offset; the difference is the lag in records. MSK also surfaces an estimate of the lag in time, which is the more actionable of the two because a thousand-record lag means something different at ten records a second than at ten thousand. Either way the shape of the curve is the diagnosis: a sudden jump is a downstream failure or a deployment, a steady climb is a consumer that cannot keep up with current throughput, and a sawtooth is a consumer that catches up between batches and is probably fine. Note that Kafka retention is by time or size, so an unread partition can be truncated by volume long before its time limit.

OpenLineage

Lineage expressed as a stream of events rather than a diagram somebody draws. The model has four pieces: a run (one execution), a job (the thing executed), datasets (inputs and outputs), and facets, extensible metadata attached to any of them — schema, column-level mapping, row counts, your own custom facet. A job emits a START event and a COMPLETE or FAIL event, and the graph is assembled from them. Amazon DataZone’s lineage feature is OpenLineage-compatible: it is captured automatically from Glue and Redshift sources added to a domain, Glue Spark job runs from version 5.0 onward can be configured to emit events to a domain, and anything else posts events to the API — which is what lets a transformation outside AWS appear in the same graph.

Cost allocation tag

A tag is only a cost dimension once it is activated in Billing and Cost Management, and activation is not retroactive: data before activation is not grouped by it. That one sentence is the whole operational lesson, because the moment you want per-pipeline cost is always after the surprise. Tag Glue jobs, Athena workgroups, Redshift clusters, EMR clusters, S3 buckets and the Lambda functions in between with a consistent key — pipeline, domain, owner — then group Cost Explorer by it. AWS-generated tags (prefixed aws:) and your own user-defined tags are activated separately. Tag policies in AWS Organizations are how you stop the key being spelled three ways, which is the other reason a tagging scheme fails.

AWS Budgets action

A budget alert tells somebody; a budget action does something. When a cost or usage threshold is breached, the action applies a specified IAM or service control policy, or stops specified EC2 or RDS instances, either automatically or after an approval. For a data platform the sane uses are narrow and worth knowing precisely: attach a deny-ish policy that blocks a sandbox workgroup from starting new Athena queries, or stop a development EMR cluster. What it is not is a per-query control — that is an Athena data usage control — and it is not a cost reduction, it is a hard stop. The forecast-based variant fires when projected spend, rather than actual spend, crosses the line, which is the only version that can act before the month is already lost.

AWS Cost and Usage Report

The raw ledger behind every other cost tool. It lands in your S3 bucket as partitioned Parquet with one row per line item per period, including resource ids and activated cost allocation tags, and the intended consumption path is Athena or a warehouse — which makes it, pleasingly, a data-engineering workload in its own right. Reach for it when Cost Explorer’s granularity runs out: cost per Glue job run, cost per Athena workgroup per day, the exact usage type behind a spike, or attributing shared storage across teams. The practical caveats are the ones any data engineer would predict — it arrives with a delay and is revised during the month, and its schema evolves, so pin your queries to named columns.

Data quality dimension

"Is the data good?" is not a question a machine can answer; six narrower questions are. Completeness — is the value present (are 2% of customer_id null today against 0.01% normally)? Uniqueness — is each entity here once? Validity — does the value satisfy its domain (is the currency code in the ISO list, is the age between 0 and 120)? Consistency — do related fields agree (does total equal the sum of the lines, does every order.customer_id exist in customer)? Timeliness — is it recent enough, the freshness axis? Accuracy — does it match reality, the only one that needs a second source to check at all, which is why reconciliation against the system of record is its usual form. Choosing which dimensions matter per column, and what threshold makes a breach, is design work; the rule language is only how you write it down.

Schema drift

The commonest production data defect, and the one whose two outcomes are both bad. A hard failure is the lucky case: the job breaks loudly, somebody looks. The soft failure is the dangerous one — a renamed column becomes all nulls, an integer silently parsed as a string sorts wrongly, a currency changed from cents to dollars makes every total a hundredth of its real value, and nothing anywhere reports an error. Three defences, in order of strength: a schema contract enforced at the boundary (ColumnExists, ColumnDataType, SchemaMatch, or an Avro/Protobuf schema in Glue Schema Registry with a compatibility mode), an open table format whose evolution rules are explicit, and at minimum a column-count and type check that fails the run. A crawler configured to update the table on every run is drift automation, not drift protection: the table dutifully follows the upstream mistake.

Late-arriving data

A mobile client that was offline, a regional feed delayed by an outage, a batch resent after a fix. The records are valid; it is the pipeline’s assumption that a day is finished at midnight that is wrong. Three responses, and the exam wants you to notice which one a scenario can afford. Drop past a stated lateness bound, which is honest only if the bound is published. Restate — reprocess the affected partition and overwrite it, which needs idempotent writes and a consumer who can cope with yesterday’s number changing. Carry forward — book the late record into the current period with a correction flag, which keeps history immutable and makes the current period slightly wrong. The design artifact that resolves it is an explicit lateness window, after which a partition is declared closed, plus a watermark that says which windows are still open.

DQDL (Data Quality Definition Language)

A ruleset is written as Rules = [ RowCount > 1000, IsComplete "customer_id", Uniqueness "order_id" > 0.99, ColumnValues "currency" in ["GBP","USD","EUR"], DataFreshness "order_date" <= 24 hours ]. Each rule is a predicate; the share that pass is the data quality score. Beyond simple rules the language carries composite rules joined by AND/OR, a where clause on many rule types so a check applies to a subset, dynamic rules that compare against the historical values of the same metric (RowCount > avg(last(5)) * 0.8) rather than a hand-set constant, and CustomSql for anything the built-in types cannot express. Cross-dataset rules exist too — ReferentialIntegrity, AggregateMatch, RowCountMatch — which is how a reconciliation against a source system becomes a rule rather than a report.

Data quality score

Glue Data Quality reports the share of rules that evaluated true. It is genuinely useful as a trend — a table whose score drifts from 100% to 92% over a month is telling you something — and it is a bad gate, because a ruleset of twenty rules scores 95% whether the failing rule was ColumnLength on a description field or IsComplete on the primary key. Gate on the rules that matter, individually: fail the job when the key is incomplete, warn when a cosmetic rule breaks. The practical arrangement is two rulesets — a blocking one with a handful of rules whose failure genuinely means do not publish, and an observational one with everything else feeding a dashboard and a weekly conversation.

Analyzer

A rule asserts; an analyzer measures. Declaring Analyzers = [ RowCount, Completeness "email", DistinctValuesCount "country" ] collects those statistics each run without any pass/fail verdict, and Glue keeps the history. That history is what makes two things possible: threshold discovery, because you can see what completeness actually is before promising 99%, and anomaly detection, because a model over the statistic’s history can flag a departure nobody wrote a rule for. AllStatistics is the broad-brush analyzer for a column. The sequencing advice follows from this: run analyzers for a couple of weeks first, read the distributions, then write rules whose thresholds reflect the data rather than an optimistic guess that will page somebody at 03:00.

Observation

The output of Glue Data Quality’s anomaly detection. Statistics collected by rules and analyzers accumulate a history; Glue models each one and surfaces an observation when a run departs from the learned pattern — a row count a third of its usual Tuesday, a completeness that stepped down and stayed there. Observations carry no verdict, which is the point: they are hypotheses for a human, and they catch the defect class rules structurally cannot, namely the one nobody predicted. Glue can also suggest a rule from an observation, which is the loop worth running — the first time a defect appears it is an observation, and after you have seen it, it becomes a rule that gates.

EvaluateDataQuality

The in-pipeline form of Glue Data Quality, and the one that makes a gate possible rather than a report. Called as EvaluateDataQuality().process_rows(frame=…, ruleset=…, publishing_options=…), it returns a collection from which you select by key: ruleOutcomes gives one row per rule with its Outcome and FailureReason, and rowLevelOutcomes gives the original rows tagged with whether they passed — which is exactly what you split on to send good rows downstream and failing rows to a quarantine prefix. publishing_options controls whether results publish to CloudWatch metrics and to the Data Quality results store (enableDataQualityCloudWatchMetrics, enableDataQualityResultsPublishing, plus an evaluation context that names the check in those metrics), and stopJobOnFailureOptions can fail the run outright. Row-level outcomes are only produced by rule types that evaluate row by row, which is why a RowCount breach can fail a job but cannot point at a row.

Deequ

Deequ computes data-quality metrics over a Spark DataFrame, verifies declared constraints against them, profiles datasets, and suggests constraints from what it finds — the mechanism behind Glue’s recommended rulesets. AWS states plainly that Glue Data Quality is built on top of it, which is why the vocabulary matches so exactly: Deequ’s analyzers, checks and constraint suggestion are Glue’s analyzers, rules and recommendations. Knowing this is worth two things in practice. It explains the shape of the managed product — metrics first, assertions over metrics second — and it names the escape hatch: when a check cannot be expressed in DQDL, Deequ (or PyDeequ) inside a plain Glue or EMR Spark job is the same engine without the wrapper. It runs on Spark by design, which is why these checks scale to billions of rows and why they cost a Spark job to run.

Write-audit-publish

The pattern that answers where does the quality gate belong. Checking after publication means consumers can read the bad rows while you decide; checking before writing means you cannot run checks that need the written form. Write-audit-publish resolves it: write to a staging location, run the ruleset against what was actually written, and publish by an atomic switch. Apache Iceberg makes the switch cheap — write to a branch, audit the branch, fast_forward the main branch to it — and the same shape works with a staging prefix plus a partition swap, or a staging table plus a view repoint. Two properties to name when a scenario asks for them: consumers see either the old data or the new data and never a half-written state, and a failed audit is a no-op rather than a cleanup.

Quarantine

The third option between fail the whole batch and let the bad rows through. Rows failing row-level validation are routed to their own prefix or table, tagged with which rule they broke and when, while the rest continue downstream — so a 0.3% defect rate does not block a daily load and the 0.3% is still available to fix and replay. Three things make a quarantine real rather than a bin nobody opens: the failure reason is stored with the row, there is a metric and an alarm on its size (an empty quarantine and a broken quarantine look identical otherwise), and there is a defined path back — reprocess into the main table idempotently once the upstream fix lands. Decide deliberately whether downstream consumers are told the batch was partial; a total that silently excludes quarantined rows is its own data-quality defect.

Data lake administrator

Lake Formation names its administrators explicitly rather than inferring them from IAM, which catches people out: a principal with AdministratorAccess is not a data lake administrator until it is listed in the Lake Formation data lake settings. Administrators can grant and revoke permissions on any database, table or view in the catalog, create data filters, register and deregister Amazon S3 locations, and view all catalog metadata regardless of grants. The practical consequence is a second, independent list of privileged identities to review — and a common operational trap, because the identity that created the catalog objects loses visibility of them the moment Lake Formation starts enforcing, unless it holds a grant or sits in that list.

AWS Lake Formation developer guide — Lake Formation personas and IAM permissions reference

Lake Formation data filter

The mechanism that lets one grant say less than "this table". A data filter belongs to a specific table and carries two independent parts: a column specification — a list of columns and nested struct fields to include or exclude — and a row filter expression, written in a subset of the PartiQL WHERE grammar (=, >, <, >=, <=, <>, BETWEEN, IN, LIKE, combined with AND/OR). How you populate the two parts decides the granularity: all-columns wildcard plus a row expression gives row-level security; named columns plus the all-rows wildcard (AllRowsWildcard, "Access to all rows" in the console) gives column-level security; named columns plus a row expression gives cell-level security. The filter is then chosen when granting SELECT to a principal, so the same physical table serves several audiences, and the engine — Athena, Redshift Spectrum, EMR, Glue — applies the filter it is handed rather than trusting the query. Created with CreateDataCellsFilter.

AWS Lake Formation developer guide — data filtering and cell-level security

Lake Formation hybrid access mode

The incremental migration path. Registering a location in hybrid access mode enforces only CREATE_TABLE, CREATE_PARTITION and UPDATE_TABLE by default; you then opt individual principals in, per database or table. An opted-in principal needs both Lake Formation permissions and IAM permissions; a principal that has not been opted in continues to work on IAM permissions alone. So one table can behave as a Lake Formation resource for the analytics team and as a plain AWS Glue resource for the legacy ETL job that nobody dares change yet. One caveat that is easy to miss: credential vending is unavailable, even for an opted-in principal, unless the underlying data location is actually registered with Lake Formation — opting in without registering grants nothing.

AWS Lake Formation developer guide — hybrid access mode

Credential vending (Lake Formation)

The mechanism that makes fine-grained access real, and the reason an audit trail looks confusing until you know about it. When a principal queries a registered location, the engine asks Lake Formation for access; Lake Formation checks the grant, then assumes the registration role you supplied when registering the location and returns temporary credentials scoped to exactly the objects the principal is entitled to. The caller never needs its own s3:GetObject permission on the data, which is the point: permissions live in the catalog, not in bucket policies. The consequence for auditing is that the resulting S3 data events are attributed to the registration role, so tracing a read back to a human needs the correlated GetDataAccess event or source identity enabled.

AWS Lake Formation developer guide — Lake Formation access control overview

Direct Amazon S3 bypass

Lake Formation filters rows and columns as data leaves the query engine. It does not encrypt or restructure the files, so a principal holding direct Amazon S3 permissions on the underlying prefix can download the Parquet file and read every row and column the filter was hiding. The filter was never breached; it was never consulted. Closing the gap is a bucket-policy job: deny s3:GetObject on the data prefix to everything except the Lake Formation registration role and the small set of roles that legitimately need raw access. The same reasoning applies one level up — a reader denied a column in the catalog but allowed to read the raw CloudTrail or application logs that quote that column has the same bypass by another route.

AWS Lake Formation developer guide — Lake Formation access control overview

GetDataAccess (the Lake Formation audit event)

The one event that answers "who queried this table". lakeformation:GetDataAccess is recorded every time an engine asks Lake Formation for temporary credentials, and it carries the requesting service, the Lake Formation principal, the table, and a role session name of the form AWSLF-<version>-<engine>-<account>-<suffix> — where the engine code is GL for an AWS Glue ETL job, AT for Athena and RE for Redshift Spectrum. Each s3:GetObject data event corresponds to one GetDataAccess call, and the shared IAM role identifier is what lets you join the two. When Lake Formation is wired to IAM Identity Center with trusted identity propagation, the event also carries the directory user id under userIdentity.onBehalfOf.userId, which is the difference between auditing a role and auditing a person.

AWS Lake Formation developer guide — logging Lake Formation API calls using AWS CloudTrail

Trusted identity propagation

Without it, a warehouse or data lake sees whichever IAM role the query tool assumed, and every audit entry reads like the same service account. Trusted identity propagation passes the Identity Center user and group identity from the client through the engine to the data service, which means Lake Formation and Amazon Redshift can grant on a directory group, Amazon S3 Access Grants can grant on a directory user, and the resulting CloudTrail events carry the user id under onBehalfOf. This is the answer whenever a requirement says permissions must follow corporate group membership, or that the audit log must identify the individual analyst — the alternative, a role per person, is the design it replaces.

AWS IAM Identity Center user guide — trusted identity propagation

Amazon S3 Access Grants

Bucket policies and IAM policies both run out of room: there is one bucket policy per bucket with a hard size limit, and a policy per team multiplies fast. Access Grants moves the mapping into its own registry — you create an instance per account and Region, register a location (a bucket or prefix) with a role that can access it, then create grants pairing a grantee with a prefix and a permission. The grantee can be an IAM principal or, once the instance is associated with IAM Identity Center, a directory user or group, which is the interesting case: the user's own identity then appears in each S3 CloudTrail data event, so "who read this object" resolves to a person without a role per person. Access Grants returns temporary credentials scoped to the grant; it does not replace bucket policies, which still act as a guardrail.

Amazon S3 user guide — managing access with S3 Access Grants

Amazon EMR runtime role

Historically every job on an EMR cluster ran with the EC2 instance profile, so that profile had to hold the union of every job's permissions — a shared cluster meant shared access to everything. A runtime role is supplied at submission time for Spark and Hive work, and the job can reach only what that role allows. Two consequences worth knowing: a step running with a runtime role cannot read the Instance Metadata Service on the cluster nodes or borrow the instance profile, which is what makes the isolation real; and Lake Formation enforcement for EMR is built on this, so the runtime role needs lakeformation:GetDataAccess plus glue:Get* in its own IAM policy — a Lake Formation SELECT grant alone fails, because the catalog call is denied before the data question is ever asked.

Amazon EMR management guide — runtime roles for Amazon EMR steps

Redshift IAM database authentication

A pipeline that stores a warehouse password has a credential to rotate; a pipeline that calls GetClusterCredentials (or GetCredentials for Redshift Serverless) has none. The caller presents its IAM identity, names a database user and optionally a set of database groups, and receives a temporary user name and password valid for a short window. The DbUser and DbGroups values can be restricted by IAM condition keys, and with AutoCreate the database user is created on first use — which is how group membership in the corporate directory becomes group membership in the warehouse. The alternative paths are Identity Center integration, which carries the actual human identity through, and a password in AWS Secrets Manager, which remains correct for third-party tools that can only speak JDBC with a static credential.

Amazon Redshift management guide — using IAM authentication to generate database user credentials

RDS and Aurora IAM database authentication

Available for MySQL, PostgreSQL and MariaDB engines. You mark a database user as using IAM authentication, grant the caller rds-db:connect on that user's resource ARN, and the client generates a token with the SDK or CLI and passes it where a password would go. The token is valid for 15 minutes — an established connection survives longer, but a new connection needs a fresh token, so connection pools and long-running jobs need to handle regeneration. The connection must use TLS, since the token travels as the password. Throughput is the usual reason not to use it: AWS documents a limit on new IAM-authenticated connections per second, so a workload that opens connections aggressively is better served by a pooled credential from Secrets Manager.

Amazon RDS user guide — IAM database authentication for MariaDB, MySQL, and PostgreSQL

MSK IAM access control

Amazon MSK offers three client authentication paths and the exam picks between them on who owns the credential. IAM access control lets a client sign with its IAM role and have topic-level actions (kafka-cluster:WriteData, ReadData, DescribeTopic, AlterGroup) authorized by IAM policy — no credential to store or rotate, and the natural choice for AWS-native producers and consumers. SASL/SCRAM stores a user name and password in AWS Secrets Manager, encrypted with a customer managed key (the AWS managed key cannot be used for this), and suits clients that cannot sign AWS requests. Mutual TLS authenticates with client certificates from AWS Private CA, and is what an existing Kafka estate with a certificate story already has. Authorization for the TLS and SCRAM paths uses Kafka ACLs rather than IAM.

Amazon MSK developer guide — authentication and authorization for Apache Kafka APIs

Redshift database role (RBAC)

Amazon Redshift roles hold privileges on schemas, tables, columns and functions, can be granted to users or to other roles, and come with system roles for common duties — sys:secadmin for security administration, sys:dba, sys:operator, sys:superuser. They matter for this domain because the warehouse's fine-grained controls attach to roles: a dynamic data masking policy is attached to a role, a row-level security policy is attached to a role, and column-level GRANT SELECT (col_a, col_b) is given to a role. So "which columns can finance see" becomes a property of one role rather than a grant per analyst, and adding an analyst is a role membership. Role membership can be driven from the corporate directory through Identity Center integration.

Amazon Redshift database developer guide — role-based access control (RBAC)

Redshift column-level access control

The simplest of the warehouse's fine-grained controls and the one people forget exists: GRANT SELECT (order_id, amount) ON orders TO ROLE analyst gives that role exactly two columns. A SELECT * by that role returns a permission error rather than a filtered row, which is the behavioural difference from dynamic data masking — masking keeps the column present and changes its values, column-level access removes the column from the role's reach. Choose on what downstream code expects: masking is transparent to a query written against the full schema, column grants are not, but column grants also guarantee no part of the value is inferable.

Amazon Redshift database developer guide — column-level access control

Redshift row-level security (RLS)

Authored with CREATE RLS POLICY … USING (predicate), attached with ATTACH RLS POLICY … ON table TO ROLE role, and switched on per table with ALTER TABLE … ROW LEVEL SECURITY ON. Managed by a superuser or the sys:secadmin role. Three behaviours decide exam questions. Policies are many-to-many — several policies on one table, one policy on several tables. All applicable policies must evaluate true for a row to be returned, so multiple policies narrow, they never widen; adding a second policy can only reduce what a role sees. And no policy means no access: a role with no applicable policy on an RLS-protected table gets zero rows, not every row, which is the fail-closed default and the usual cause of "the report went empty after we enabled RLS". RLS composes with column-level access control, so row and column restrictions apply together.

Amazon Redshift database developer guide — row-level security

Redshift dynamic data masking (DDM)

CREATE MASKING POLICY defines an expression over the column's value — full redaction, a partial reveal of the last four digits, a hash, or a call to a tokenisation function — and ATTACH MASKING POLICY … ON table(column) TO ROLE role PRIORITY n binds it. Four mechanics are examinable. Several policies can attach to one column, and when a querying identity matches more than one, the highest priority applies per column; the default priority is 0. You cannot attach two policies to the same column at equal priority, nor two to the same role-and-column pair. TO PUBLIC sets the default everyone gets unless a higher-priority attachment covers them, which is how you make redaction the fallback and visibility the exception. And the masking happens on the way out: the base table still holds the real values, so DDM controls reading, not storage, and anyone who can bypass the policy — a superuser, or a principal reading an unloaded copy in Amazon S3 — sees the original.

Amazon Redshift database developer guide — dynamic data masking

Amazon Redshift datashare

A datashare names schemas, tables, views and functions and exposes them to a consumer that queries them in place — no unload, no copy, no second storage bill, and the consumer always reads current data. Within an account the producer adds consumer namespaces directly. Across accounts there is a two-way handshake: the producer administrator authorizes the datashare, then the consumer administrator associates it with the whole account or with named clusters and workgroups, or declines it. Two requirements catch people: for cross-account sharing both the producer and the consumer cluster must be encrypted, and objects in a datashare are read-only unless the producer explicitly grants write privileges such as INSERT — with cross-account writes needing a separate authorization on the producer side and a specific association on the consumer side. Datashares are also the clean answer to "share without granting access to the production cluster", since a producer can share from a separate namespace fed by a copy.

Amazon Redshift database developer guide — sharing data across AWS accounts

The cross-account data chain

Nothing about a cross-account read is a single permission. A reader in account B querying a table owned by account A must satisfy, in effect: an identity policy in B allowing the action; a resource policy in A naming B as a principal — a bucket policy, a Glue Data Catalog resource policy, a Lake Formation grant made through AWS RAM; a catalog grant if Lake Formation is enforcing, since IAM alone cannot express a column filter; and the KMS key policy of the customer managed key the data is encrypted with, because decryption is a separate authorization from retrieval. The failure signature differs usefully. A missing resource policy or grant produces AccessDenied on the data call. A missing key grant produces AccessDenied from KMS, or a read that lists objects and fails on content — the shape of error that makes people blame replication or networking. The key policy is the link most often forgotten, because it is in a different console from everything else.

AWS Lake Formation developer guide — cross-account data sharing

AWS Glue Data Catalog resource policy

The Data Catalog takes a resource-based policy exactly as a bucket does, and only one policy at a time can be attached to a catalog, so cross-account grants accumulate as statements in a single document that quickly becomes the hardest-to-review object in the account. It governs metadata operations — glue:GetTable, GetDatabase, GetPartitions — which is a separate question from whether the caller can read the underlying objects: granting catalog access without the corresponding Amazon S3 or Lake Formation permission produces a reader who can see schemas and read nothing. When Lake Formation manages the catalog, its own cross-account grants (fulfilled through AWS RAM) are the better path, because they carry the fine-grained filters a resource policy cannot express.

AWS Glue developer guide — resource-based policies within AWS Glue

Lake Formation cross-account version

Lake Formation has revised how it shares across accounts several times, and the account-level CROSS_ACCOUNT_VERSION parameter selects which behaviour you get. The practical difference is where the grant lives: the earliest version required a matching AWS Glue Data Catalog resource policy alongside every Lake Formation grant, while later versions make the grant through AWS RAM alone — the recipient accepts a resource share, and named-principal grants to accounts in the same AWS Organizations organization are accepted automatically. The version matters in two places on a real estate: a grant that silently does nothing is usually a version mismatch, and upgrading is a deliberate migration because the old resource-policy statements have to be cleaned up afterwards.

AWS Lake Formation developer guide — cross-account data sharing

Bucket owner enforced (S3 Object Ownership)

Cross-account writes used to leave the writing account owning the object, which produced the classic data lake fault: a partner lands a file in your bucket, and your own analytics role cannot read it. Object Ownership set to bucket owner enforced ends that by disabling access control lists altogether — every object is owned by the bucket owner regardless of who wrote it, and ACL grants are rejected rather than ignored. It is now the default for new buckets, so this is the behaviour you get unless someone chose otherwise. The migration cost is real on an old estate: anything that relied on an ACL (a legacy log-delivery configuration, a cross-account bucket-owner-full-control dance) must move to bucket policy first, and the documented prerequisite is to migrate bucket and object ACL permissions to policies and reset the bucket ACL to private before flipping it — S3 Inventory is the tool for finding ACL use at scale.

Amazon S3 user guide — controlling ownership of objects and disabling ACLs

AWS Glue Data Catalog encryption at rest

Schemas leak. A table named patient_claims with columns ssn, diagnosis_code and settlement_amount describes sensitive data without exposing a single row, which is why metadata gets its own encryption setting. Enabled once per account in the Data Catalog settings with a symmetric KMS key, it covers databases, tables, partitions, table versions, connections and user-defined functions. The operational consequence is the examinable one: every client that reads the catalog now needs KMS permission as well as glue:Get*, so enabling it on a live account breaks crawlers, jobs and query engines whose roles were never granted on the key. Priced into the same decision is kms:Decrypt call volume, since catalog reads are frequent.

AWS Glue developer guide — encrypting your Data Catalog

Glue connection password encryption

A Glue connection holds the credentials a job uses to reach a JDBC source, and by default the password sits in the connection properties. Turning this setting on encrypts the password with a named KMS key as part of CreateConnection or UpdateConnection, storing it in an ENCRYPTED_PASSWORD property and returning it encrypted thereafter. Two details matter. It is independent of metadata encryption — you can enable password encryption alone, which is the cheap first move on an existing catalog. And any principal that creates or updates a connection then needs kms:Encrypt on that key, so a pipeline that provisions its own connections needs the grant. Clearing the setting leaves previously encrypted passwords encrypted under the key they were written with, which is a quiet dependency on a key you must not delete.

AWS Glue developer guide — encrypting connection passwords

AWS Glue security configuration

Catalog encryption protects metadata; a security configuration protects what a job produces. It names up to three independent encryption settings — Amazon S3 data written by the job, the CloudWatch Logs the job emits, and the job bookmark, the stored progress marker that can itself contain key values from the source data — each with its own KMS key. Attach it to a job or crawler and the role passed to that job must hold permission on every key named, which is the usual cause of a job that fails immediately with a KMS error after a configuration is added. Bookmark encryption is the one people skip and the one an auditor asks about, because a bookmark holding the maximum value of a customer identifier column is derived sensitive data.

AWS Glue developer guide — setting up encryption in AWS Glue

Redshift cluster encryption

Worth knowing precisely, because the reflex answer is wrong. You can modify an unencrypted Amazon Redshift cluster to use AWS KMS encryption with an AWS owned key or a customer managed key, and Redshift performs the migration itself: during the operation the cluster is available read-only and reports status resizing. The reverse (encrypted to unencrypted) is possible through the CLI but not the console. Two exceptions remain: hardware security module encryption cannot be enabled by modifying a cluster — that needs a new HSM-encrypted cluster and a data migration — and cross-Region snapshot copy must be disabled before changing encryption. Contrast Amazon RDS, where encryption really is creation-time only and the migration path is a snapshot copied with encryption and then restored.

Amazon Redshift management guide — changing cluster encryption

require_ssl (and sslmode)

Two halves of one control, and a question can turn on either. Server side, require_ssl is a parameter group setting on an Amazon Redshift cluster: set it to true and the cluster refuses any connection that is not using TLS. Because it lives in a parameter group, the change applies on a reboot, which is why "we set it and nothing changed" is a common report. Client side, the JDBC or ODBC sslmode decides what TLS actually buys: require encrypts but does not verify the server certificate, so it does not protect against an impostor endpoint, while verify-ca and verify-full check the certificate chain and, for verify-full, the host name. A requirement phrased as "protect against man-in-the-middle" needs verify-full, not merely require_ssl. The equivalent elsewhere: an Amazon RDS parameter such as rds.force_ssl, the "Require SSL connection" property on an AWS Glue JDBC connection, and aws:SecureTransport for Amazon S3.

Amazon Redshift management guide — configuring security options for connections

aws:SecureTransport

Amazon S3 accepts both HTTP and HTTPS, so a bucket is not TLS-only because you intend it to be. The enforcement is a bucket policy statement with Effect: Deny on s3:* when aws:SecureTransport is false — an explicit deny, which no later allow can override. Two refinements that appear in questions. The deny should cover the bucket and its objects (arn:aws:s3:::bucket and arn:aws:s3:::bucket/*), because a policy naming only objects leaves bucket-level calls unprotected. And TLS version is a separate condition key, s3:TlsVersion, for requirements phrased as "must use TLS 1.2 or later" — aws:SecureTransport is a yes/no about encryption, not a statement about the cipher suite or protocol version.

Amazon S3 user guide — enforcing encryption in transit

MSK encryption in transit

Amazon MSK encrypts between clients and brokers, and separately between brokers in the cluster. Broker-to-broker encryption is a simple on or off. Client-to-broker has three values: TLS only, plaintext only, or TLS and plaintext, and the third is the one that matters for an audit — the cluster still accepts unencrypted connections on a separate port, so a "TLS enabled" cluster can be carrying cleartext traffic from the one legacy producer nobody migrated. The setting is chosen at cluster creation. Note the interaction with authentication: IAM access control and SASL/SCRAM both require TLS, so selecting either forecloses the plaintext option for those listeners.

Amazon MSK developer guide — Amazon MSK encryption

Partial redaction

The most common masking output and the easiest to reason about wrongly. Showing ****1234 lets a support agent confirm which card a customer means, which is a genuine business requirement that full redaction cannot meet. The cost is that the revealed fragment is real data: for a value with little entropy outside the shown fragment, or where the hidden part is derivable (a national identifier whose leading digits encode a birth date and region), partial redaction can be close to no protection at all. It is also not a join key — two rows with ****1234 are not evidence of the same card. Implemented in the warehouse with a dynamic data masking expression, or upstream with a transformation in the pipeline.

Amazon Redshift database developer guide — dynamic data masking use cases

Tokenisation

The option that preserves the most analytical capability while removing the sensitive value from the analytics estate. A token is a meaningless surrogate — often format-preserving, so a 16-digit card becomes 16 digits and downstream schemas do not change — mapped back to the original only through a vault that very few principals can reach. Because the mapping is consistent, joins work: two tables tokenised with the same vault still join on customer, COUNT(DISTINCT token) is the true distinct count, and a cohort analysis is unaffected. Reversibility is the trade: tokenisation is pseudonymisation, not anonymisation, so the data remains personal data for regulatory purposes and the vault becomes the thing you must protect, audit and back up. Range and ordering are lost, so a token cannot answer "amounts above a threshold" — that needs the real value or a separately derived band.

AWS Prescriptive Guidance — data tokenization patterns on AWS

Irreversible hash (with salt)

Hashing looks like the strong option and sometimes is, but the properties need stating. A deterministic hash preserves equality, so it joins and counts exactly like a token, and it needs no vault, which removes an operational burden. It is not reversible by design — but "not reversible" depends entirely on entropy and salt: an unsalted SHA-256 of a national identifier, a card number or an email address is recovered by hashing the candidate space, which for most identifier formats is small enough to exhaust. A shared secret salt or a keyed HMAC fixes that, at the cost of reintroducing a secret to manage and the rule that every system that must join on the value needs the same salt. The decisive limitation is that there is no legitimate path back: when a support case or a regulator genuinely needs the original, a hash cannot produce it and a token can.

AWS Prescriptive Guidance — data anonymization and pseudonymization techniques

Pseudonymisation vs anonymisation

The distinction decides whether a privacy obligation still applies to a dataset, which makes it a design question rather than a legal footnote. Pseudonymisation replaces identifiers with surrogates while keeping a path back — a token vault, a salt, a key — so the data remains personal data, deletion requests still reach it, and residency rules still bind it. Anonymisation removes that path such that re-identification is not reasonably possible, which in practice means more than deleting the name column: unique combinations of postcode, birth date and job title identify individuals perfectly well, so genuine anonymisation needs aggregation, generalisation or suppression of the quasi-identifiers too. The pipeline consequence: a "anonymised" analytics table that retains one row per person and a hashed id is pseudonymised, and treating it as out of scope is the mistake that gets found in an audit.

AWS Prescriptive Guidance — data anonymization and pseudonymization techniques

AWS Glue sensitive data detection

The in-pipeline counterpart to Macie. The Detect Sensitive Data transform scans either every column or a sample, matching managed entity types (names, addresses, national identifiers, card numbers, credentials) and custom patterns, and then takes an action in the job: report the finding to a column, redact the value, partially mask it, or replace it with a hash. Because it runs inside the ETL job it can stop the sensitive value from ever reaching the destination, which is strictly stronger than masking at the destination and is the right answer when a requirement says the data must not be written. The cost is scan time proportional to the detection coverage you choose, which is why the sample-based mode exists.

AWS Glue developer guide — detect and process sensitive data

Macie automated sensitive data discovery

The mode to reach for when the question is "which of our buckets hold personal data" rather than "what exactly is in this bucket". Once enabled, Macie samples objects continuously across the account — or across every member account, when run from the delegated administrator — and builds an inventory with a sensitivity score per bucket. Scores start at 50, move between 1 and 99 as evidence accumulates (up when sensitive data is found, down when it is not), and read -1 when Macie could not analyse the bucket at all, which is itself a finding worth chasing: an unsupported object type, or a KMS key Macie has no permission on. What it inspects is configured by a sensitivity inspection template naming the managed data identifiers, custom data identifiers and allow lists to use. It samples rather than scanning exhaustively, which is the deliberate cost trade and the reason it complements rather than replaces a discovery job.

Amazon Macie user guide — performing automated sensitive data discovery

Macie sensitivity score

A triage signal, not a risk rating. The score reflects how much data Macie inspected, how many sensitive data types it found, how many occurrences of each, and the nature of those types — so a high score means "there is probably sensitive data here", not "this bucket is business-critical". Every bucket begins at 50 (unknown) and moves as sampling proceeds; -1 is the distinct state meaning unable to analyse. Individual detection types can be suppressed from a bucket's score, which is how an expected and accepted finding stops dominating the dashboard. Treat the score as the thing that tells you where to point a real discovery job, and the per-object findings as the evidence.

Amazon Macie user guide — assessing your Amazon S3 security posture

Macie sensitive data discovery job

Where automated discovery samples broadly, a discovery job inspects what you point it at: selected buckets or a bucket criteria expression, run once or on a daily, weekly or monthly schedule, with the managed and custom data identifiers you choose and an optional allow list. Output is a sensitive data finding per affected object, naming the bucket, the object, the identifier that matched, the number of occurrences and, when configured, the location of the match in the object — written to a results bucket you own and encrypted with your key. Pricing is by bytes inspected, which is the whole reason the sampling mode exists. And the limit that decides exam questions: Macie reports, it never modifies, redacts, moves or deletes an object. Remediation is your pipeline's job — typically an EventBridge rule on the finding driving a quarantine or a masking step.

Amazon Macie user guide — running sensitive data discovery jobs

Custom data identifier

Managed data identifiers cover the formats the world shares: card numbers, national identifiers, keys and tokens, health and financial patterns across many countries. They know nothing about your internal customer reference, your policy number scheme or a partner's account format, and that gap is where most real coverage is missing. A custom data identifier is a regular expression, optionally narrowed by keywords that must appear within a maximum match distance of the match, and by an ignore list of known-benign values such as test fixtures. The keyword-proximity mechanism is the part worth understanding: a bare six-digit pattern matches far too much, while the same pattern required to sit near the word "policy" is precise. The same concept appears in CloudWatch Logs data protection policies, so one definition of "what our sensitive identifiers look like" can be applied at rest and in logs.

Amazon Macie user guide — building custom data identifiers

Macie allow list

Findings nobody acts on train everyone to ignore findings. An allow list is the mechanism for the ones that are genuinely fine: a list of literal strings (the sample card numbers in a test fixture, the organisation's own published phone numbers) or a regular expression describing a benign pattern. Macie checks matches against the allow list and excludes them from findings and from sensitivity scores. The discipline it enforces is useful — adding a value to an allow list is an explicit, reviewable decision with an owner, which is a different thing from a dashboard that everyone has learned to scroll past.

Amazon Macie user guide — defining sensitive data exceptions with allow lists

Basic event selector

A trail configured with basic event selectors can log data events for exactly AWS::S3::Object, AWS::Lambda::Function and AWS::DynamoDB::Table, choose read-only, write-only or all, and name up to 250 resources across all selectors on the trail. What it cannot do is the limitation that forces the upgrade: it cannot filter on eventName, and it cannot reach any other resource type — including objects in S3 directory buckets. So "log only DeleteObject on this one bucket", the natural cost-control request, is not expressible here. Advanced event selectors supersede it for every new design; basic selectors matter mainly because existing trails have them and a question may hinge on what an existing trail can be asked to do.

AWS CloudTrail user guide — logging data events

Advanced event selector

Advanced event selectors build an expression over fields: eventCategory, resources.type, resources.ARN (with StartsWith and NotStartsWith for prefix matching), eventName, readOnly, and more. Two things follow. First, reach: every data-event resource type beyond the original three is available only here, which includes directory-bucket objects and the long list of service resource types AWS keeps extending. Second, cost: data events are charged per event and a busy data lake generates them in the millions, so narrowing to the prefixes and event names that answer your actual audit question is the difference between an affordable trail and a surprising bill. The caution AWS itself gives: for auditing purposes record all data events, and use narrow selectors for operational monitoring — the selector that saves money is also the selector that loses the evidence you did not predict needing.

AWS CloudTrail user guide — logging data events with advanced event selectors

SET_SOURCE_IDENTITY (Lake Formation)

Credential vending means the S3 data events for a governed table are attributed to the Lake Formation registration role, not to the analyst who ran the query — so every read looks identical and the audit question "who read this object" has no answer from the S3 events alone. Setting SET_SOURCE_IDENTITY to TRUE in the data lake settings, and adding sts:SetSourceIdentity alongside sts:AssumeRole in the registration role's trust policy, makes STS propagate a source identity that appears in the sessionContext of each s3:GetObject event. It requires S3 data events to be enabled on the bucket, which is a separate switch and a separate charge. Without this, the correlation path is the GetDataAccess event joined to the S3 event on the role identifier — which works, and is more work.

AWS Lake Formation developer guide — logging Lake Formation API calls using AWS CloudTrail

Redshift database audit logging

Not on by default, and the enabling sequence is examinable. The connection log records authentication attempts, connections and disconnections with client address and authentication method; the user log records changes to database user definitions; the user activity log records every query before it runs. All three are switched on together through the console, API or CLI — but the user activity log additionally requires the database parameter enable_user_activity_logging to be true, which is false by default. Enable the feature without the parameter and you get connection and user logs only, which is the classic "we turned on audit logging but there are no queries in it". Parameter-group changes take effect on reboot. Provisioned clusters can export to Amazon S3 or CloudWatch Logs; Redshift Serverless exports to CloudWatch only. Logging to the STL system tables is automatic and not optional, but those tables retain only a few days of history, which is why the exported logs exist.

Amazon Redshift management guide — database audit logging

Redshift STL and SYS system views

Redshift logs its own activity to system tables and views whether or not audit logging is enabled: STL_QUERY and SYS_QUERY_HISTORY for executed SQL, STL_CONNECTION_LOG for sessions, SVL_ views for aggregated detail. This is why "who ran this query yesterday" is often answerable with a SELECT and no configuration at all. The catch, and the reason the exported audit log is not redundant, is retention: system tables hold a rolling window of a few days and are trimmed as the cluster works, so they cannot serve a compliance requirement to retain query history for months or years. The pattern that satisfies both is system views for day-to-day investigation and audit logging to CloudWatch or Amazon S3 for the retained record.

Amazon Redshift database developer guide — system tables and views reference

Athena query history

Athena records each execution and exposes it through the console and through ListQueryExecutions plus GetQueryExecution, which returns the statement text, the submitting principal, the workgroup, the output location, the data scanned and the engine time. It answers questions CloudTrail cannot: CloudTrail logs that StartQueryExecution happened and by whom, but the statement and the bytes scanned live here. Retention is limited, so a requirement to keep query history for an audit period means exporting it — commonly a scheduled job writing executions to a table, or EventBridge on query state changes. Note the division of labour for a governed lake: Athena history says what was asked, GetDataAccess says what was authorised, and S3 data events say what bytes moved.

Amazon Athena user guide — working with query results and recent queries

Data residency

Customer data in a Region stays in that Region unless you move it — but a data pipeline has many ways to move it: a cross-Region replication rule, a snapshot copy, a global table, a cross-Region datashare, a Firehose destination in another Region, a disaster-recovery job nobody reviewed. Intention is not a control, so residency is implemented preventively with service control policies on aws:RequestedRegion, declarative or Control Tower Region-deny controls, and Config rules that report resources outside the allowed set. Two caveats that turn into questions. Global services have a home Region — IAM and Organizations data lives in us-east-1, and some services store configuration metadata centrally even when the data plane is Regional, so "no metadata may leave the EU" is a stricter claim than "no data may leave the EU" and is not always satisfiable. And residency is not the same as sovereignty, which adds requirements about who can operate and access the infrastructure.

AWS whitepaper — addressing data residency with AWS

Deletion request (right to erasure)

The requirement that exposes whether a data platform was designed or accreted. Deleting a row from the warehouse is the easy part; the question is where else the value went — the raw landing zone, the curated Parquet partition, a materialised view, a Redshift copy, an OpenSearch index, a BI extract, the CloudTrail and application logs that quoted the identifier, the token vault, and every backup and snapshot taken since. Four design decisions make it tractable rather than heroic. A table format with row-level deletes (Iceberg, Hudi, Delta) so removing one record does not mean rewriting a partition. Lineage recorded well enough to enumerate derivatives. Tokenisation at ingest, so deleting the vault entry severs the link everywhere at once — the single highest-leverage choice available. And a retention schedule that expires copies on a clock, so the set of places to search is bounded. Note the honest tension with immutability: a record under S3 Object Lock in compliance mode cannot be deleted before its retain-until date by anyone, so a legal-hold obligation and an erasure obligation have to be reconciled in the retention design rather than at request time.

AWS Prescriptive Guidance — data privacy patterns on AWS

Row-level delete (open table formats)

A plain Hive-style table over Parquet in Amazon S3 has no notion of a row-level delete — removing one record means reading the file it lives in, writing a new file without it, and swapping them, which for a day-partitioned dataset can mean rewriting gigabytes to erase one person. Open table formats change the arithmetic: Apache Iceberg supports DELETE FROM and UPDATE statements, tracking the change through metadata and delete files so only the affected data files are touched, and Athena, Amazon EMR, AWS Glue and Amazon Redshift can all issue them. This is a privacy capability as much as an engineering one, which is why it appears in this domain: it is the difference between an erasure request that takes a query and one that takes a batch job and a maintenance window. Expect to pair it with periodic compaction, since accumulated delete files slow reads.

Amazon Athena user guide — using Apache Iceberg tables

Retention schedule

Retention is a two-sided obligation and most platforms implement only one side. Keeping data long enough is remembered. Not keeping it longer than necessary shows up in a privacy audit, in a discovery request that sweeps ten years of records nobody needed, and in the storage bill. Enforced automatically, it is a small set of mechanisms: S3 Lifecycle expiration (including noncurrent-version expiry and delete-marker cleanup), CloudWatch Logs group retention, Kinesis and MSK retention periods, AWS Backup plan lifecycles, and a table-format expiry job for warehouse data. The interaction worth knowing: a lifecycle expiration cannot delete an object version still under an Object Lock retention period or legal hold, and that precedence is exactly why the two combine safely — Object Lock holds the record for the required term, a lifecycle rule expires it shortly after, so the obligation ends on schedule instead of becoming a permanent archive.

Amazon S3 user guide — managing the lifecycle of objects

Data classification

Classification is the mechanism that turns security from a per-dataset negotiation into a lookup. A workable scheme has few levels, an unambiguous assignment rule, and a named control set per level: who may grant access, whether a customer managed key is required, whether cross-Region copying is allowed, what retention applies, whether masking is mandatory outside a named role. On AWS the label itself has somewhere to live and something to enforce it — tags on buckets and tables, LF-Tags in the catalog driving Lake Formation grants, DataZone metadata forms, and Config rules or SCPs keyed on the tag value. Macie is how you check the label is true: a bucket tagged internal that automated discovery scores highly for personal data is a classification failure, and that comparison is the most useful report in a governance programme.

AWS whitepaper — data classification: secure cloud adoption

DataZone subscription

The workflow that gives federated governance its audit trail. A consumer project requests access to a published asset; the producer approves or rejects, optionally with terms; on approval DataZone fulfils the grant for Lake Formation-managed AWS Glue tables and Amazon Redshift tables and views, and emits an event for anything else. What makes it governance rather than a ticket queue is that the request, the approver and the reason are recorded against the asset, so the answer to "why does this team have this data" exists without archaeology. The design consequence worth noting: DataZone does not replace Lake Formation permissions or bucket policies — it drives them, so a grant made directly in Lake Formation is invisible to the catalogue and is exactly the back channel a governance programme needs to close.

Amazon DataZone user guide — working with subscribed data

Data steward (and data owner)

Every governance mechanism in this domain assumes someone can answer "may this team have this data". Without a named owner per dataset the answer defaults to whoever is least likely to say no, and subscription queues stall. The distinction in common use: the owner is accountable for the dataset and its access decisions, usually in the producing business area; the steward does the ongoing work — classification, glossary definitions, quality expectations, reviewing access. On AWS the role is expressed concretely rather than on an org chart: project membership and ownership in Amazon DataZone, a data lake administrator or grantor in Lake Formation, an owner tag on the bucket and table, and the approver on the subscription. The point is not the title but that the access decision has an address.

Amazon DataZone user guide — Amazon DataZone terminology and concepts