Kinesis Data Streams: shards and ordering

Lesson 1 of 5 in Streaming at Scale.

A shard is two things at once, and almost every Kinesis mistake comes from treating it as one.

It is a capacity unit. Each shard accepts up to 1 MB per second or 1,000 records per second of writes, whichever it reaches first, and serves 2 MB per second of reads shared across all classic consumers. On small records the records limit binds long before the bytes limit — at 200 bytes a shard is full at 200 KB/s, a fifth of its nominal byte capacity — which is why record size belongs in every sizing calculation.

It is also an ordering domain. Records are sequenced within a shard and nowhere else. The stream as a whole has no order.

Those two roles meet in the partition key. Every record carries one; Kinesis hashes it with MD5 into a 128-bit value and routes the record to the shard whose hash range contains it. So two consequences follow that are tested relentlessly.

The ordering you actually get is per key. "Keep one customer's events in order" is satisfied by customerId. "Keep every event in the stream in order" is satisfied only by a single shard, which caps you at 1 MB/s — and if a stem asks for total ordering at high throughput, the honest answer is that the two requirements are in conflict and one must give.

Your capacity is only reachable if the keys spread. A key with few values, or a skewed distribution, concentrates traffic into one hash range. The stream then reports plenty of spare capacity in aggregate while producers are refused — see hot shard, which is the single most common Kinesis incident and the one whose obvious fix is wrong.

The part of that simulation worth stating in words is why resharding does not fix a hot key, because it is the answer most people give.

Resharding has two primitives: SplitShard divides one shard's hash range into two children, and MergeShards combines two adjacent ranges. UpdateShardCount performs a series of those to reach a target count — provisioned mode only, and it recommends a target that is a multiple of 25% of the current count because any other target takes longer to reach.

A split halves a range. A single partition key hashes to a single point. So that point falls in exactly one half, and after the split you have two shards where one is still saturated and the other is empty — and you are paying for both. The repair has to happen on the producer: raise the key's cardinality by salting it (key#random(0..N)), making it composite, or choosing a different field.

And the salt has a cost that a good answer names: ordering holds only within one partition key, so region salted into region#0..15 means no single consumer can read one region's events in order again. You have traded ordering scope for a flat load. If the ordering was load-bearing, the correct answer is a different key, not a salted one.

Three limits on resharding that make reactive autoscaling a trap: ten scaling operations per rolling 24 hours per stream, no scaling beyond double or below half the current count in one call, and a ceiling of 10,000 shards without a quota increase. A genuinely spiky day cannot be tracked inside ten changes, which is why on-demand capacity mode exists — and note that on-demand still partitions by key, so it does not rescue a skewed one.

Key terms: Shard, Partition key, Hot shard, Resharding, On-demand capacity mode, Enhanced fan-out, Parallelization factor

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.