Streaming ingestion: Kinesis, Firehose, Flink, Kafka
Lesson 3 of 5 in Ingesting and Storing Training Data.
When data is being born — clicks, swipes, sensor readings — you ingest a stream, and the exam gives you four names to keep straight. Kinesis Data Streams is the durable, ordered event log you build on. Amazon Data Firehose is the no-code delivery service that lands streams in S3 and friends. Managed Service for Apache Flink is the stateful processing engine that computes over streams. Amazon MSK is managed Apache Kafka for teams already speaking Kafka.
Two renames to absorb before reading anything older: Firehose dropped its "Kinesis" prefix in 2024, and "Kinesis Data Analytics" became Managed Service for Apache Flink in 2023. Current exam questions use the new names.
| Service | You build/manage | Latency & ordering | Replay | Reach for it when |
|---|---|---|---|---|
Kinesis Data Streams | Producers and consumers (KCL, Lambda, Flink); shards or on-demand capacity | Sub-second reads; ordering per partition key within a shard | Yes — re-read anywhere in retention (24 h default, extendable up to a year) | Multiple independent consumers, strict per-key order, reprocessing after bugs |
Amazon Data Firehose | Nothing — fully managed, serverless delivery | Near-real-time: buffers by size or time before each write | No — it delivers; it is not a store | The easiest road to S3/Redshift/OpenSearch, with optional in-flight Lambda transform and conversion to Parquet/ORC |
Managed Service for Apache Flink | A Flink application (SQL, Java, Python) | Real time, with event-time windows and watermarks | Replays from its sources (Kinesis/MSK); holds checkpointed state, not the stream | Stateful transforms: sliding-window aggregates, stream joins, sessionization |
Amazon MSK | Kafka topics, producers, consumers (brokers managed; serverless option) | Low latency; ordering per partition | Yes — topic retention you configure | Kafka-native teams, Kafka ecosystem tooling, migrations from self-run Kafka |
The shape of the choice: Firehose when the requirement is "get events into S3, converted and batched, with nothing to operate." Kinesis Data Streams when several consumers read the same stream, order per key matters, or you may need to rewind and reprocess. Flink is not storage at all — it sits on a stream and computes (counts per 10-minute window, joins of two streams). MSK versus Kinesis is less about capability than ecosystem: existing Kafka producers, connectors, and habits point to MSK; starting fresh on AWS, Kinesis is the lower-ops default.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.