Infrastructure as code for pipelines

Lesson 2 of 4 in Programming Concepts on the Exam.

Infrastructure as code appears on this exam for one reason that matters more than the syntax: a data pipeline's resources are mostly stateful, and a careless update deletes data rather than reconfiguring it.

CloudFormation is the engine — JSON or YAML templates deployed as stacks. CDK gives you a real programming language whose synth compiles constructs into a CloudFormation template. SAM is a CloudFormation extension with short-hand resource types for serverless applications plus a CLI that builds, packages and runs them locally. Terraform is the multi-cloud alternative, with its own state file rather than a managed stack.

The mechanism to understand properly is the change set. Submitting a template directly applies it. Creating a change set instead produces a list of resource-level actions with no changes made, which you then execute or discard. The column to read is Replacement: True means CloudFormation will create a new resource and delete the old one — and for a database, an EBS volume or anything stateful, that means the data is gone.

Reviewing that column is the practice that separates a routine deployment from a Friday-evening incident, and it is why change sets exist at all.

Three neighbouring mechanisms a question will test alongside it. Drift detection compares deployed resources with the template and reports what somebody changed by hand — a template is only the truth if nobody edited the console. Stack policies deny update actions on named resources, which is a hard guard where review is a soft one. And a deletion policy of Retain or Snapshot on a stateful resource is what prevents a stack deletion from taking the data with it.

For a data pipeline the resources to protect are the obvious ones — an S3 bucket, a Glue Catalog database, a Redshift cluster, a Kinesis stream — and the property changes that force replacement on each are worth knowing before a release rather than during one.

The four infrastructure-as-code options, by the thing that actually decides between them. All of the first three deploy CloudFormation stacks, so change sets and drift detection apply to all three.
	Authoring	Reach for it when	What you give up
CloudFormation	YAML or JSON templates	the estate is AWS, the team wants no build step, and the template is the artefact under review	loops, conditionals and abstraction — repetition is copied rather than generated
CDK	TypeScript, Python and others; `synth` emits a template	there is real repetition — one stack per environment, per table, per account — or shared constructs across teams	the deployed template is generated, so reviewing a diff means reviewing synthesised output as well as code
SAM	a CloudFormation template with `Transform: AWS::Serverless-2016-10-31`	the stack is mostly serverless and the local build, package and `sam local invoke` workflow is the point	expressiveness outside the serverless resource types, where you drop back to raw CloudFormation anyway
Terraform	HCL, with its own state file	the estate is multi-cloud, or the organisation has already standardised on it	managed stack state and native drift detection — the state file becomes an artefact you must secure, lock and back up

Which pipeline resources are dangerous to replace

Replacement is harmless for a Glue job, a Lambda function or a Step Functions state machine — they hold no data. It is destructive for anything that does, and the properties that force it are worth knowing before a release.

An S3 bucket. Changing BucketName replaces it, which means a new empty bucket and an attempt to delete the old one (which fails if it is not empty, leaving the stack in a failed state). Use DeletionPolicy: Retain.

A Kinesis stream. Changing Name replaces it, destroying records still in retention and every consumer's position.

A Redshift cluster or RDS instance. Several properties force replacement; DeletionPolicy: Snapshot is the minimum protection, and a stack policy denying replacement is better.

A Glue Data Catalog database or table. Replacing the database takes its tables with it, and the data in S3 survives while the metadata that made it queryable does not.

A DynamoDB table. Changing the key schema replaces the table, and the items are gone.

The habit: deploy through a change set, read the Replacement column, and put a stack policy on every resource whose replacement would be a data-loss incident.

Key terms: AWS CloudFormation, Change set, AWS CDK, AWS SAM