When two services both fit
Lesson 4 of 4 in The Data Service Map.
Here is the uncomfortable truth about this exam: for most scenarios, two of the four options would work. That is not sloppy item writing — it is the job. Real estates contain two services that do overlapping work, and a data engineer earns their keep by knowing which one the constraint picks.
Which means a question with two workable options is not asking "which is better". It is asking "which clause in this stem separates them" — and for every genuinely competing pair there is exactly one such clause. Learn the pair as a separator, not as two feature lists, and these questions collapse from a judgement call into a lookup:
Both of these move streaming data durably. The separator is whether the requirement names the Apache Kafka API. It does. Done.
The four pairs below account for most of the two-both-fit questions on this exam. Memorise the middle column.
| The pair | Why they look interchangeable | The one constraint that decides | Which way it falls |
|---|---|---|---|
Kinesis Data Streams vs Amazon MSK | Both are durable, partitioned, replayable streams that several consumers can read independently, both scale to millions of records per second, and both can retain records for days. | Does the requirement need the Apache Kafka API and ecosystem? | Kafka named, existing Kafka clients, Kafka Connect connectors, or a migration from a self-managed Kafka cluster → MSK. Anything else, especially "least operational overhead" or native triggering of a function → Kinesis Data Streams. |
AWS Glue vs Amazon EMR | Both run Apache Spark at scale over data in the lake, both read and write the shared catalog, and both will happily chew through terabytes on a schedule. | Do you need the cluster itself? | A named ecosystem tool beyond Spark, node-level control, custom libraries on the nodes, Spot capacity, or sustained heavy use where a long-running cluster is cheaper → EMR. A scheduled transformation with the least operational overhead and nothing to size → Glue. |
Amazon Athena vs Redshift Spectrum | Both run SQL over data left in place in object storage, both read external table definitions from the shared catalog, and neither requires loading the data first. | Does the query have to join lake data to warehouse tables? | One query spanning external files and tables already loaded in the warehouse, or users who already live in the warehouse → Spectrum, which borrows the warehouse’s compute. Ad hoc querying with nothing running between queries → Athena. |
AWS Step Functions vs Amazon MWAA | Both express dependencies between steps, both retry what is retryable, both branch on failure, and both will orchestrate the same set of data jobs. | Is Apache Airflow itself part of the requirement? | Existing Airflow graphs to migrate, workflows defined in Python, a specific Airflow provider, or a team already running Airflow → MWAA, which runs continuously and therefore costs money while idle. A serverless state machine with no idle cost and direct service integrations → Step Functions. |
Two services both fit — which one does the stem want?
Interactive decision tree — outcomes:
- Amazon MSK
Managed Apache Kafka: real Kafka brokers, real Kafka protocol, so existing producers, consumers, Kafka Connect connectors and Streams applications work unchanged. That compatibility is the only reason to prefer it here — if the stem never mentions Kafka, choosing MSK buys you broker and topic management you did not need.
- Amazon Kinesis Data Streams
A durable, shard-partitioned stream with retention from 24 hours up to a year, so many consumers can read the same records independently and a consumer can be rewound to replay them. Ordering is guaranteed within a shard, which is per-partition-key rather than global — a detail that matters the moment a stem says "in order".
- Amazon Data Firehose
A delivery stream: buffer records by size or time and write them to a managed destination with no code. It is the least-effort answer and the one that gives up the two things a stream gives you — there is no retention to replay from and no second independent consumer. Short buffer intervals also manufacture the small-files problem, which is the follow-up question.
- Amazon EMR
A managed cluster for the whole Hadoop and Spark ecosystem — Hive, HBase, Trino, Flink and the open table formats — with control over instance types, Spot capacity and what is installed on the nodes. It wins on named tooling and on sustained heavy use, where a long-running cluster is cheaper than per-run serverless capacity; it loses whenever the stem says there should be nothing to size or patch.
- AWS Glue
Serverless Spark with no cluster to provision, integrated with the shared catalog, and with job bookmarks so a scheduled job can process only what is new. The default answer for a scheduled transformation — and the right answer almost any time "least operational overhead" appears next to a batch job.
- AWS Lambda
Event-driven, short-lived compute: ideal for a few seconds of work per arriving object — validate it, enrich it, route it, start something. The hard edge is the 15-minute maximum execution time, which is why a Lambda-shaped answer to a multi-gigabyte distributed transformation is a distractor rather than a design.
- Amazon Athena
Serverless SQL over data in place, billed by bytes scanned, with nothing running between queries. The natural answer for ad hoc analysis over the lake — and the service whose cost questions are really format and partitioning questions.
- Redshift Spectrum
External tables read from object storage by the warehouse, so a single query can join cold history in the lake to hot dimensions already loaded in warehouse tables. It uses the warehouse’s compute, which is both the advantage and the condition: there has to be a warehouse.
- Amazon OpenSearch Service
A search and log-analytics engine: free-text relevance, fast filtering over semi-structured documents, and log exploration. It is not a warehouse substitute — an aggregate over three years of fact rows belongs in the warehouse or the lake, and that is the boundary a stem tests by describing either searching or aggregating.
- Amazon MWAA
Managed Apache Airflow: workflows as Python code, the Airflow provider ecosystem, and existing graphs that run unchanged. The environment runs continuously, so it costs money while idle — which is exactly the trade a stem rules in or out when it talks about cost at rest.
- AWS Step Functions
A serverless state machine: states, branches, per-state retry and catch, direct integrations with services so steps need no glue code, and no cost while nothing is running. The default orchestrator unless the requirement names Airflow.
- Amazon EventBridge
A router and a scheduler rather than an orchestrator: it starts things when an event matches a rule or when the clock says so. If the stem has no dependency graph — "process each file as it lands", "run this at 02:00" — then reaching for a workflow engine is over-building.
Interactive sorting exercise: Each card is a clause from a stem. Put it with the service that clause picks. Nothing here is decided by preference — every card contains its own separator.
| The pair | The separator | Why it is worth knowing |
|---|---|---|
Kinesis Data Streams vs Data Firehose | Does anything need to read the records more than once? | A delivery stream buffers and writes to a destination; it keeps no retention for you to replay and supports no second independent consumer. Pick it when "land it with minimal effort" is the whole requirement, and reject it the moment replay or a second consumer appears. |
AWS DMS vs AWS DataSync | Is the source a database or a file system? | Database replication, including ongoing change capture and heterogeneous engine conversion, is the migration service. Bulk file and object transfer between on-premises storage and AWS storage is the sync service. Stems describe one or the other unambiguously, and candidates still swap them. |
AWS Lambda vs AWS Glue | How long does one unit of work take? | Seconds of work per arriving object is a function; minutes to hours of distributed work over large data is a job. The 15-minute function timeout is the hard edge that makes this a factual question rather than a stylistic one. |
Glue Data Catalog vs Lake Formation | Metadata or permission? (These are not alternatives.) | The catalog says what tables exist, what columns they have and where the partitions are. Lake Formation decides who may read which tables, columns and rows. A stem about a column one team must not see is a permission question even though the column is defined in the catalog. |
Amazon OpenSearch vs Amazon Redshift | Searching documents or aggregating facts? | Free-text relevance, log exploration and fast filtering over semi-structured documents on one side; SQL aggregation and joins over large fact tables on the other. The giveaway verbs in a stem are search and investigate against aggregate and report. |
The trap in each of the four main pairs
Kinesis against MSK. The trap is cost or throughput reasoning. Both scale enormously, and a stem that invites you to compare them on records per second has not given you the separator. Look for the word Kafka — or its fingerprints: consumer groups, topics, Connect, a client library.
Glue against EMR. The trap is assuming serverless always wins on cost. At sustained high utilisation a long-running cluster with Spot capacity is frequently cheaper; serverless wins on operational overhead, which is a different axis the stem has to name.
Athena against Spectrum. The trap is that both read external tables from the same catalog, so they look like the same product with different front doors. The separator is whether warehouse-resident tables are in the query — and the hidden condition is that Spectrum needs a warehouse to run on at all.
Step Functions against MWAA. The trap is complexity. "Twelve interdependent steps" sounds like it demands Airflow, and a state machine handles it perfectly well. The separator is Airflow itself: existing graphs, Python-defined workflows, a specific provider.
What to do when the stem genuinely gives you no separator
It happens, and there is a default order that is right more often than it is wrong.
- Prefer the managed or serverless option unless the stem names something it cannot do. "Least operational overhead" is the most common qualifier on this exam, and even when it is absent, AWS exams reward the answer with fewer components you own.
- Prefer the purpose-built service over the general one. A delivery stream over a function that polls, a quality ruleset over hand-written checks, a catalog over a configuration file.
- Prefer the option that does not move the data. Querying in place beats copying, because a copy is a second thing to secure, govern and keep fresh.
- Reject anything that introduces a component the stem did not ask for. An extra queue, an extra account, a cluster nobody mentioned — each is a plausible-sounding addition and a reliable sign of a distractor.
Why "both would work" is the right instinct to have
A candidate who finds two workable options has read the stem correctly. The failure is not noticing two — it is picking between them on taste, which is a coin flip dressed as expertise.
So when two survive, say out loud: what is the clause that separates these? If you can name it, you have the answer and the reason. If you cannot find one, go back to the stem, because a well-built item put it there — most often as a number with a unit, a named technology, or a phrase about what must not change.
Key terms: Replayability, Bytes scanned, Partitioning, The small-files problem, Lakehouse, Deciding constraint
Interactive checkpoint quiz (3 questions) — open this page in a browser to take it.