MWAA and Airflow DAGs
Lesson 3 of 5 in Orchestrating the Pipeline.
Amazon MWAA is real Apache Airflow with the scheduler, metadata database, web server and an auto-scaling worker pool managed for you. Your DAGs, plugins.zip and requirements.txt live in an Amazon S3 bucket that MWAA synchronises, so deployment is an S3 upload — which is also why a bad requirements.txt can break the whole environment rather than one DAG.
Three things genuinely justify choosing it, and a stem names one of them.
The provider ecosystem. Hundreds of third-party systems have operators and sensors already written and tested — a Snowflake query, a Salesforce extract, a Slack notification, an HTTP endpoint. Reaching those from Step Functions means writing and owning the client.
Backfill as a first-class operation. Airflow formalises a run's period as its data interval, so airflow dags backfill is a command rather than a project. That shows up in a stem as "re-run the pipeline for last March".
DAGs that need real code. Shared helpers across twenty DAGs, graph structure computed at parse time, a factory that generates one task per table in a config file. All of that is natural in Python and awkward in a state machine.
And the three costs, stated plainly. The environment is billed while it exists, not per execution, so an idle environment has a floor. You own Airflow's operational surface: version upgrades, worker sizing, a scheduler that can fall behind, and a metadata database that grows until somebody prunes it. And a DAG is code, so it needs the testing and review discipline code needs — a DAG file with a syntax error stops the scheduler parsing it, and an expensive top-level API call runs on every parse, not once per run.
Our scheduler keeps falling behind. Where do we look?
Three places, in order, and the first is the one people miss.
Top-level code in DAG files. Every DAG file is parsed on a short interval, so a module-level database query or API call runs continuously in the scheduler process. A DAG factory that reads a config table at parse time is the classic version. Move the expensive work inside the task.
Sensors in poke mode. They hold worker slots, so the queue backs up while the UI reports tasks as running. Switch long waits to reschedule, or to a deferrable operator that uses the triggerer.
Environment class and worker count. Only after the first two, because scaling up an environment that is wasting its capacity on parsing and poking just wastes more expensively.
And one maintenance item that creeps: the metadata database grows with every task instance and log entry, and a large one slows the scheduler and the UI. Pruning it is a routine operational task, not an emergency measure.
When should a step be an Airflow operator rather than a Lambda function the DAG invokes?
Use the operator when one already exists for the system, because it has handled the authentication, pagination, retry semantics and status polling that you would otherwise write. That is the ecosystem argument for MWAA, and spending a week re-implementing a provider is the mistake it exists to prevent.
Use a Lambda function when the logic is yours: a business rule, a transformation, a bespoke validation. Then the DAG's job is orchestration and the function's job is the work, which keeps the DAG readable and the logic testable outside Airflow.
What to avoid in both directions: business logic inside a DAG file (it runs at parse time and cannot be unit-tested easily), and a DAG that is one enormous PythonOperator doing everything (you have an Airflow-shaped cron job with no visibility into which step failed).
MWAA or Step Functions for a nightly five-step AWS-only pipeline?
Step Functions, and the deciding axis is cost at idle rather than capability.
Five states running once a night is a handful of state transitions — a cost so small it rounds to nothing — and the .sync integrations wait for Glue and EMR without a polling loop. An MWAA environment is billed while it exists, so the same pipeline carries an environment-hour floor every hour of every day whether or not it runs.
MWAA earns its floor when there are many pipelines, third-party systems in the mix, or a genuine need for backfill and Python. One small AWS-only pipeline is not that case, and recognising it is a frequent exam judgement.
Key terms: Amazon MWAA, Airflow operator, Airflow sensor, Backfill, DAG (directed acyclic graph)