Amazon MSK and the Kafka model
Lesson 2 of 5 in Streaming at Scale.
Amazon MSK runs open-source Apache Kafka, and that is the whole proposition: your existing producers, consumers, Kafka Streams applications and Kafka Connect plugins work unchanged, and so does the operational vocabulary. AWS owns broker provisioning and patching, the metadata layer and the CloudWatch integration; you still own topic configuration, partition counts and client tuning.
The honest comparison with Kinesis Data Streams is not about capability. Both are durable, ordered, replayable logs that scale to whatever you need. It is about what you are optimising for, and three things point at MSK: API compatibility with an existing Kafka estate, the connector ecosystem, and per-topic configuration that Kinesis simply does not expose. Three point the other way: a smaller operational surface, native integration with Lambda and Firehose, and capacity you never size.
Where the models genuinely differ is worth knowing precisely, because that is what a question tests.
Partition count only goes up. You can increase a topic's partitions; nothing decreases them. An over-partitioned topic is a permanent cost in open file handles, replication traffic and metadata.
Adding partitions breaks key affinity. The partition is chosen by hash modulo the partition count, so raising the count re-maps existing keys. Records for one key can be read out of order across the change unless you drain first — a discontinuity Kinesis does not have, because a split preserves the hash range relationship between parent and child.
Durability is yours to configure. Replication factor, min.insync.replicas and producer acks together decide how many broker failures you survive and whether an acknowledged write can be lost. Kinesis replicates across three Availability Zones with no setting to get wrong.
Consumption parallelism is capped by partition count. Within one consumer group a partition goes to exactly one member, so a 6-partition topic read by 10 consumers leaves four idle.
| Kinesis Data Streams | Amazon MSK | |
|---|---|---|
Unit of ordering and capacity | shard — 1 MB/s or 1,000 rec/s in, 2 MB/s out | partition — throughput depends on broker sizing |
Changing the count | split or merge in both directions; ten operations per rolling 24 hours per stream | increase only, never decrease — and raising it re-maps existing keys |
Fan-out to many consumers | shared 2 MB/s and five | any number of consumer groups, each with its own offsets, sharing broker bandwidth |
Long retention | 24 h default, 7 days extended, up to 365 days with long-term retention | tiered storage, or broker disk you provision |
Authentication | IAM only | IAM, SASL/SCRAM via Secrets Manager, or mutual TLS with AWS Private CA |
Native AWS integrations | Lambda, Firehose, Managed Flink, DMS target, CloudWatch — all first-class | Lambda, Firehose and Managed Flink too, plus the whole Kafka Connect ecosystem via MSK Connect |
Reach for it when | the stem says "minimise operational overhead", there is no existing Kafka, or the consumers are AWS services | the stem names Kafka — a migration, existing Kafka Streams code, an existing connector, or SCRAM and mTLS clients |
MSK Serverless or provisioned?
Serverless removes broker sizing: you create a cluster, create topics, and per-partition throughput is managed for you, billed per cluster-hour, partition-hour and GB.
The three things that rule it out, each of which a stem can state. Authentication is IAM only — clients using SASL/SCRAM or mutual TLS need provisioned. Per-partition throughput ceilings and partition-per-cluster quotas are tighter, so a very large steady workload belongs on provisioned. And broker-level configuration is not yours to set, which matters if the application depends on a specific topic or broker property.
The usual reading: unpredictable or spiky traffic and no appetite for sizing brokers → serverless. Large, steady, SCRAM or mTLS clients, or custom configuration → provisioned.
When is MSK Connect the right answer rather than a stream processor?
When the work is moving data rather than computing on it, and a connector already exists.
Kafka Connect moves data between Kafka and everything else with configuration instead of code: a source connector pulls from a database or queue into a topic, a sink connector pushes a topic into S3, OpenSearch or a warehouse. MSK Connect runs those same community or custom plugins as a managed, auto-scaling connector, so you upload the plugin and a configuration.
Its limit is the boundary of single-message transforms: field renames, routing, masking a value, dropping a record. It cannot join two topics, maintain a windowed aggregate or sessionise — that is where Managed Service for Apache Flink starts. A stem mentioning an existing Debezium change-capture connector, or an S3 sink the team has used for years, is pointing at MSK Connect; one mentioning a five-minute rolling average is not.
I need to add partitions to a busy topic. What breaks?
Key affinity, and only across the boundary.
The partition is hash(key) % partitionCount, so changing the count changes the mapping for most keys. Records for key k written before the change are in the old partition; records written after are in a new one. A consumer reading both partitions has no ordering relationship between them, so for a key whose records straddle the change, order is not guaranteed.
If ordering per key is load-bearing, the safe sequence is: stop producing, let consumers drain to the end of every partition, add the partitions, then resume. If that is impossible, the alternative is a new topic with the desired partition count and a migration of producers and consumers over to it — which is why choosing the partition count up front matters more in Kafka than choosing a shard count does in Kinesis.
Key terms: Amazon MSK, Kafka partition, Consumer group, MSK Connect, MSK Serverless, MSK tiered storage