Replay, retention, and the late consumer
Lesson 5 of 5 in Streaming at Scale.
Retention looks like a cost setting and is actually an architecture decision, because it bounds three different things at once.
It bounds replay. A transformation bug found on Tuesday makes every downstream record derived since the bug wrong, and the only honest repair is to recompute from the source — which requires the source to still exist. If your only copy of the truth is a 24-hour stream, you have a 24-hour statute of limitations on your own bugs.
It bounds recovery. A consumer that is down, or merely slow, for longer than retention has a permanent hole. Nothing errors; the records are simply deleted before anyone reads them.
It bounds onboarding. A new consumer starting at the oldest available record can only see as far back as retention allows, so "give the new team the last month" is a retention question before it is an access question.
The settings: Kinesis defaults to 24 hours, IncreaseStreamRetentionPeriod raises it to as much as 7 days at an extended rate, and long-term retention keeps data for up to 365 days with its own per-GB charges for reads older than 7 days. MSK tiered storage is the Kafka equivalent. Firehose has none at all.
Which is why the standard production shape is two components rather than one: a stream for the latency and a raw landing zone in Amazon S3 for the truth, because S3 retention is effectively unlimited and the raw zone becomes the replay source once the stream's window has closed.
Iterator age is rising. Finding the cause, in order.
- IteratorAge climbing
The consumer is reading records older and older. Nothing has errored — this is the failure mode that is silent until the data is gone.
- Is the function erroring?
Check the consumer’s own error rate alongside iterator age.
- A poison record is blocking the shard
An ordered shard retries the failing batch in place, so nothing behind it progresses. Set BisectBatchOnFunctionError to isolate the bad record, bound MaximumRetryAttempts, and add an on-failure destination — which receives metadata about the skipped batch, not the records, since those are still in the stream.
- Is the lag on every shard or one?
Compare per-shard iterator age. One shard behind while the others are current is a distribution problem, not a capacity one.
- Fix the partition key
One shard carrying most of the traffic means one consumer doing most of the work. Raising the parallelization factor will not help, because records for one key stay in one batch group. Raise the key’s cardinality instead — and accept the narrower ordering guarantee.
- Where is the consumer’s time going?
Look at duration per record, not per invocation, and at what the function waits on.
- It waits on something downstream
An external call in the hot path, or a sink that cannot absorb the write rate. The remedy is batching the downstream write, caching the lookup, or raising the sink’s capacity — the stream is not the constraint, and no stream setting addresses it.
- It is simply doing the work slowly
Raise the parallelization factor (up to 10 concurrent batches per shard, grouped by partition key so per-key order survives), raise the batch size to amortise invocation overhead, or make the per-record work cheaper. This is the cheap case: no resharding, no producer change.
- How much runway is left?
Compare iterator age with the retention period. That difference is the deadline, and it is the only number that matters while you diagnose.
- Buy time first, then fix
IncreaseStreamRetentionPeriod applies going forward and widens the window for records still in the stream, so it can prevent further loss while you work. It cannot recover records already deleted — retention is not retroactive.
| Starting position | Kinesis | Kafka | Use it when |
|---|---|---|---|
Oldest available record |
|
| onboarding a new consumer that must see all retained history, or a full reprocess |
Only new records |
|
| a consumer that genuinely does not care about the past — and the default that silently skips the backlog |
A specific point in time |
| offsets looked up for a timestamp | targeted recovery: reprocess from the moment the defect shipped |
A specific record |
| an explicit offset per partition | resuming from a checkpoint your own application stored |
Where this consumer left off | the KCL lease table in DynamoDB | the committed group offset | normal operation — and note that in Kinesis the position lives in your lease table, so deleting it restarts the consumer |
Key terms: Stream retention period, Replay, Iterator age, Backfill, Dead-letter queue