Firehose: managed delivery and its limits

Lesson 3 of 5 in Streaming at Scale.

Firehose is the answer to a question people rarely ask plainly: how do I get records into the lake, continuously, with nobody on call? There is no shard count, no consumer to write, no checkpoint and no scaling policy. You configure a source, a destination, two buffering numbers and optionally a transform, and the service delivers.

The numbers are the lesson, because every one of them has a question built on it.

Buffering. For Amazon S3, buffer size is 1–128 MB and buffer interval 0–900 seconds, and Firehose flushes when either is reached. So the interval is a latency ceiling and the size is a file-size floor, and together they are the dial that decides whether you get the small-files problem downstream. A 60-second interval on a quiet stream produces 1,440 objects a day per prefix; 128 MB / 900 seconds produces a handful of well-sized ones at up to fifteen minutes of delay.

What turning on the good features costs you. Enabling Parquet or ORC conversion or dynamic partitioning raises the minimum buffer size to 64 MB, with 128 MB the default. Zero buffering is unavailable with dynamic partitioning. And with dynamic partitioning Firehose buffers in several internal stages, so end-to-end delay can be about 1.5 times the configured interval — a pipeline promised "60-second freshness" can legitimately deliver in 90.

The ceiling that produces a real incident. Each active partition holds its own buffer, and there are 500 active partitions per delivery stream. Past that, the overflow records go to the S3 error prefix with activePartitionExceeded — data in the wrong place rather than data lost, which is a distinction worth making in an answer.

And one behaviour that is easy to miss: when delivery falls behind ingestion, Firehose raises its own buffer size to catch up. So the file sizes you observe in production can exceed the ones you configured, and that is the service protecting durability rather than a misconfiguration.

Which streaming component a requirement admits. Read down the column for the requirement the stem states most precisely — the first "no" is the answer.
Requirement	Firehose alone	Data Streams	Data Streams + Firehose
Land records in S3 with no code	yes — this is its purpose	no — you write a consumer	yes
Reprocess last week after a bug	no — nothing is retained	yes, within the retention window	yes, plus the raw S3 copy
Several independent consumers	no — one source, one destination	yes, with enhanced fan-out past two or three	yes
Sub-second end-to-end latency	no — the floor is the buffer interval	yes, especially with enhanced fan-out	yes, on the stream path
Per-key ordering	no — a buffer makes no ordering promise	yes, per partition key	yes, on the stream path
Convert to Parquet on the way in	yes, against a Catalog table	no — your consumer does it	yes
Nothing to size or scale	yes	only in on-demand mode	on-demand, yes

Three Firehose numbers that are the whole of a question

When a stem gives you a freshness requirement and a Firehose configuration, check these before anything else.

Conversion or dynamic partitioning raises the minimum buffer size to 64 MB. A design that specified a 5 MB buffer and then enabled Parquet conversion is not the design that will run.
Zero buffering is not available with dynamic partitioning. "Deliver immediately, partitioned by customer" is not a configuration that exists.
With dynamic partitioning, end-to-end delay can reach about 1.5x the configured interval, because of multi-stage internal buffering. A 60-second interval does not guarantee 60-second freshness; it guarantees roughly 90.

The fourth, which is not a number: Firehose raises the buffer size itself when delivery falls behind ingestion. If a stem reports objects much larger than configured, that is the service catching up, not a misconfiguration.

Key terms: Amazon Data Firehose, Firehose buffering hints, Dynamic partitioning, Record format conversion