Throttling, retries, and backpressure

Lesson 4 of 4 in Ingestion Patterns.

Every stage of a pipeline has finite capacity, so sooner or later arrival exceeds service. At that moment there are exactly three things a system can do, and reading which one it does tells you how it will fail.

Push back. Refuse the work and make the caller's problem. Kinesis Data Streams does this: a write past a shard's 1 MB/s or 1,000 records/s gets a ProvisionedThroughputExceededException, and it is the producer's job to try again. So does every throttled AWS API.

Buffer. Accept everything and hold it. Amazon SQS does this: the queue grows, ApproximateAgeOfOldestMessage climbs, and nothing is lost until retention expires.

Drop. Accept the work and discard what does not fit. A fire-and-forget producer with no retry does this, and it is the only one of the three that loses data without saying so.

The reason this taxonomy earns its place is that it makes a widespread mistake visible. A buffer is not a fix. It converts a throughput problem into a latency problem and buys time, and it becomes a fix only if the average arrival rate is below capacity and the burst is finite. If the average exceeds capacity, every buffer in the world postpones the same failure and makes it bigger — and in the meantime the age of the oldest message is growing, which is a service-level failure of its own.

Two things in that simulation are worth saying in words, because they are the parts that get answered wrongly.

Backoff without jitter is barely better than no backoff. If a thousand throttled clients each sleep 100 ms, then 200 ms, then 400 ms, they come back in synchronised waves and are refused together — the service sees the same spike it just rejected. Jitter is the random component that spreads the cohort out, and it is what actually lets the backlog drain. The AWS SDKs include it in their default standard retry mode; adaptive goes further and adds a client-side rate limiter that measures the throttling it is receiving and slows the client before it is rejected.

The fix for throttling is usually not a quota increase. Walk the diagnosis in order: is the load genuinely above total capacity, or is it concentrated? For Kinesis, enhanced shard-level monitoring answers that — WriteProvisionedThroughputExceeded on one shard with the others idle is a hot shard, and the repair is the partition key, not more shards. For an API, is the client retrying immediately, or retrying a non-retryable error forever? And for a small-record producer, is the binding limit the records-per-second one rather than the bytes one — in which case KPL aggregation can remove the problem entirely by packing many user records into one Kinesis record.

The partial failure that hides inside an HTTP 200

PutRecords sends up to 500 records in one call. If some of them are throttled, the call returns HTTP 200 with a FailedRecordCount and a per-record error code — because the call succeeded even though some records did not. A producer that checks only the status code therefore loses data silently, and the loss scales with how throttled the stream is.

The correct pattern is to inspect the response, re-submit only the failed records, and apply backoff with jitter between attempts. The same shape appears in DynamoDB's BatchWriteItem (UnprocessedItems) and BatchGetItem (UnprocessedKeys), and in SQS's SendMessageBatch. No SDK retry mode covers it, because from the SDK's point of view nothing failed.

If a stem describes a producer that "reports success" while records are missing, this is almost always the mechanism.

Diagnosing throttling — in the order that finds the cause fastest

Producers are being throttled
ProvisionedThroughputExceededException, a 429, or a rising FailedRecordCount.
Is the load concentrated or spread?
Enable enhanced shard-level monitoring on the stream. Stream-level totals cannot answer this — a hot shard is precisely the case in which the aggregate looks fine.
One shard hot, others idle
WriteProvisionedThroughputExceeded concentrated on one shard while the stream total sits well under capacity.
Fix the partition key
Raise cardinality: salt it, make it composite, or choose a different field. Resharding cannot help — a split halves a hash range, and one key hashes to one point. The cost is a narrower ordering guarantee, and a good answer says so.
Spread evenly — which limit is binding?
Compare incoming bytes against 1 MB/s per shard and incoming records against 1,000/s per shard. On small records the records limit binds long before the bytes limit.
Records limit binding, small records
KPL aggregation packs many user records into one Kinesis record up to the 1 MB ceiling, so the shard counts the container rather than the contents. Consumers must deaggregate — the KCL and a Lambda event source mapping do it transparently, a hand-written GetRecords consumer does not.
Genuinely over capacity
Both limits are at their ceiling across every shard: the stream needs more capacity, not a better key.
Reshard, or move to on-demand
UpdateShardCount for a known steady load. For a spiky one, note the default limit of ten scaling operations per rolling 24 hours per stream, which reactive autoscaling cannot work around — on-demand mode is the service’s own answer.
Is the client making it worse?
Check the retry policy before changing any capacity: an immediate-retry client raises offered load above its own send rate.
Fix the retry policy first
Exponential backoff with jitter, a bounded attempt count, and re-submission of only the failed records from a batch response. Consider adaptive retry mode for a bulk job. This is free and often resolves the symptom on its own.

The same overload, three transports. What grows is the thing to alarm on, and it differs — which is why one monitoring template does not fit all three.
Transport	Behaviour at overload	What grows	Alarm on	Data loss when
Kinesis Data Streams	pushes back per shard	rejected writes, and the producer’s own retry backlog	`WriteProvisionedThroughputExceeded`, plus `FailedRecordCount` in the producer	the producer gives up, or an unread record passes the retention window
Amazon SQS	buffers without limit	queue depth, and the age of the oldest message	`ApproximateAgeOfOldestMessage` — depth alone can be healthy at high throughput	retention expires (maximum 14 days), or a message exceeds `maxReceiveCount` with no dead-letter queue
Amazon Data Firehose	buffers internally and raises its own buffer size to catch up	delivery delay, and the file sizes you observe	`DeliveryToS3.DataFreshness`, and records written to the S3 error prefix	essentially never for S3 delivery — it retries and then writes failures to the error prefix, which is data in the wrong place rather than data gone
Lambda on a stream	retries the failing batch in place, in order	iterator age — and it blocks every record behind the failure	`IteratorAge`, and the function’s error rate	a poison record is retried until the data ages out, unless `BisectBatchOnFunctionError`, `MaximumRetryAttempts` and an on-failure destination are set
A plain HTTP producer	drops	nothing observable	there is nothing to alarm on, which is the finding	immediately, and silently — the only detection is a count that does not reconcile

Key terms: Backpressure, Exponential backoff with jitter, ProvisionedThroughputExceededException, KPL record aggregation, Visibility timeout, Dead-letter queue

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.