Processing a stream in flight

Lesson 4 of 5 in Streaming at Scale.

An unbounded stream has no last record, so any computation over it is a snapshot of a thing still arriving. That forces a question batch never has to answer: which records count?

The answer is a window. A tumbling window is a fixed, non-overlapping slice — "orders per minute", where each record belongs to exactly one window. A sliding (hopping) window advances by less than its length, so records belong to several — "the five-minute average, updated every minute". A session window groups records separated by less than a gap of inactivity, so it has no fixed length at all.

Then comes the question that separates a correct answer from a plausible one: which clock?

Event time is when the thing happened. Processing time is when your code saw it. They differ whenever arrival is delayed, and the difference is not cosmetic. A mobile client goes through a tunnel and uploads twenty minutes of buffered events at once. Windowed by processing time, all of them land in the current minute and that minute reports a spike that never occurred. Windowed by event time, each goes to the window its timestamp belongs to and the numbers are right — but the earlier windows had already closed and reported, so the system must either keep them open longer or correct itself.

There is no free option, and the deciding phrase is usually in the requirement. An operational dashboard counting what the system is doing now legitimately uses processing time. Billing, auditing, session analysis or anything reconciled against another system must use event time, because the other system knows when the event happened.

Event-time windowing needs one more mechanism, and it is the one worth understanding properly: a watermark.

An event-time window faces an impossible question — when may it emit a result, given that a late record could always turn up? A watermark answers it by fiat. The engine tracks the maximum event timestamp it has seen, subtracts an allowed lateness, and declares that windows ending before that point are complete and may fire.

So the watermark is a tuning knob on a correctness-versus-latency trade, not a fact about the data. Generous allowed lateness means accurate windows that report late and operator state held in memory longer. A tight one means fast results and dropped stragglers. Apache Flink lets you route late records to a side output rather than discarding them, which is how a pipeline can be both prompt and eventually correct: emit on the watermark, then reconcile the side output into the stored result.

And here is the judgement call a stem is usually testing. If a small percentage of events arrive hours late and must still be counted, that is not a windowing problem — hours of allowed lateness is not operable, because the state would be enormous and every result delayed. It is a two-path design: stream for the prompt approximate answer, batch recomputation over the raw landing zone for the authoritative one. Recognising that the stated lateness exceeds what a watermark can sensibly absorb is the whole question.

Key terms: Tumbling window, Event time vs processing time, Watermark, Checkpointing and savepoints, Amazon Managed Service for Apache Flink, Exactly-once processing