Streaming at Scale

Kinesis Data Streams, Managed Service for Apache Kafka, and Firehose as three different bargains: who owns the shard or partition, how ordering is guaranteed and at what granularity, what happens when a consumer falls behind, and when a fully managed delivery stream is the right answer instead.

Content current as of 2026-10.

Lessons

1. Kinesis Data Streams: shards and ordering
2. Amazon MSK and the Kafka model
3. Firehose: managed delivery and its limits
4. Processing a stream in flight
5. Replay, retention, and the late consumer

Sources

• Amazon Kinesis Data Streams developer guide — resharding, scaling and parallel processing
• Amazon Kinesis Data Streams developer guide — developing enhanced fan-out consumers
• Amazon Managed Streaming for Apache Kafka developer guide
• Amazon Data Firehose developer guide — buffering hints
• AWS Lambda developer guide — using Lambda with Amazon Kinesis Data Streams