Streaming or batch — the deciding constraints
Lesson 1 of 4 in Ingestion Patterns.
Nobody gets a question that says "would you like streaming or batch?" What you get is a paragraph about a business, and the answer is already determined by one sentence in it. Your job is to find that sentence.
The mistake is to decide on feel — streaming sounds modern, batch sounds like a mainframe — when the decision is a property of the data and the requirement, and each has a crisp test.
Is the data bounded? Can you name the moment the input is complete? Yesterday's orders: yes, at midnight. A clickstream: no, never. If you cannot name that moment, either you are streaming or you are inventing a boundary — a nightly cutoff — and accepting the late records it strands.
What is the latency requirement, in units somebody will be held to? Not "real time", which means nothing. "The fraud decision must return before the payment is authorised" is a hundred-millisecond budget. "Analysts look at yesterday each morning" is a six-hour batch window with all night to run in. Between those extremes sits the micro-batch band — one to fifteen minutes — which is where most pipelines calling themselves streaming actually live, and where Firehose with a generous buffer is both cheaper and simpler than a stream processor.
The reason this ordering matters is that a latency requirement is a budget you can spend. Five minutes of allowed staleness buys you batching, which buys you larger files, columnar formats, cheaper compute and simpler failure handling. A hundred milliseconds buys you none of those, and you pay for all of it in operational complexity.
| Constraint | Points at streaming when… | Points at batch when… | Why it decides |
|---|---|---|---|
Latency the business is held to | a decision, alert or user-visible value depends on the record within seconds | the consumer is a person reading a dashboard on a cadence, or a downstream job with its own schedule | latency is the one requirement you cannot meet by spending more later; everything else can be retrofitted |
Boundedness of the source | the source never stops and has no natural end — devices, events, a database change log | the source produces a complete artefact — a nightly extract, a vendor file, a table snapshot | an unbounded source forces you to choose which records count, so windows and watermarks enter the design whether you wanted them or not |
Replayability needed | several independent consumers, or a requirement to reprocess recent history after a bug | the source itself is durable and re-readable — the files are still in the bucket | a transport that forgets (Firehose, Amazon SQS after deletion) cannot answer "run last Tuesday again"; a log (Kinesis Data Streams, Apache Kafka) can, for as long as its retention allows |
Shape of the load | arrival is continuous, and the peak-to-average ratio is modest | arrival is one large lump, or so spiky that provisioning for the peak is absurd | a continuously running stream processor costs money while idle; a batch job costs nothing between runs but pays a cold start every time |
Cost of being wrong by a few minutes | a stale answer causes a bad decision — fraud, pricing, capacity, safety | a stale answer is merely stale, and the correction lands in the next run | this is the question that separates "we would like it fresher" from "it must be fresher", and the second one is the only one worth the complexity |
Which ingestion shape does this requirement need?
Interactive decision tree — outcomes:
- A stateful stream processor
Amazon Managed Service for Apache Flink, or Spark Structured Streaming on Glue or EMR. You are buying windowing, event-time semantics and checkpointed state — and taking on a late-data policy, a snapshot strategy and a continuously running application.
Sanity check before you commit: if the "state" is a simple count per key over a window of 15 minutes or less, a Lambda consumer with a tumbling window on the event source mapping does it with no cluster at all.
- A Lambda consumer on the stream
Per-record enrichment, routing, validation or a write to a serving store, with no cluster to run. The event source mapping brings the controls that matter: batch size, a batching window, parallelization factor up to 10 per shard, and the error handling —
BisectBatchOnFunctionError,MaximumRetryAttempts, an on-failure destination — without which one poison record blocks its shard until the data ages out. - Amazon Data Firehose into Amazon S3
The cheapest correct answer in the minutes band. Set buffering hints for the file size you want, turn on format conversion to Parquet against a Glue table, and add dynamic partitioning if the prefix must come from the record rather than from arrival time.
Remember the two numbers that change when you do: minimum buffer size rises to 64 MB, and with dynamic partitioning the end-to-end delay can be about 1.5x the configured interval.
- A Kinesis data stream, with Firehose as one of its consumers
This is the shape to reach for when a stem mentions several teams, or reprocessing. The stream is the durable, replayable log; Firehose subscribes to it and handles the S3 delivery; other consumers read the same records independently, with enhanced fan-out if there are more than two or three of them.
You have bought replay and fan-out, and you now own a retention decision and an iterator age alarm.
- Change data capture, not a scheduled query
AWS DMS full-load-plus-CDC, a zero-ETL integration if the source and target happen to be a supported pair, or a Debezium connector on MSK Connect for a Kafka estate. The deciding word in the stem is deletes: a
WHERE updated_at > :last_runquery cannot see a deleted row, and no amount of scheduling fixes that. - AWS DataSync on a schedule
Managed, verified, incremental transfer between on-premises storage and AWS storage, with encryption in transit and integrity checking. Reach for it when the phrase is "every night" or "keep in sync", and the network can carry the delta.
- AWS Snow Family
Do the arithmetic before offering this, because the arithmetic is the justification: a saturated 100 Mbps link moves about 1 TB per day, so 500 TB over a dedicated 1 Gbps circuit is roughly 50 days of continuous saturated transfer. Past that point a courier is faster, and the data is encrypted with AWS KMS keys on a tamper-evident device.
If the requirement is ongoing rather than one-time, the answer is Direct Connect or DataSync instead — Snow solves a backlog, not a flow.
- AWS Transfer Family
A managed SFTP, FTPS, FTP or AS2 endpoint writing straight into S3, because the partner will not change protocol for you. Pair it with a S3 event notification to start the pipeline, and consider a Transfer Family managed workflow for the decrypt-and-decompress step that a partner file usually needs.
Key terms: Bounded and unbounded data, Stream retention period, Replay, Firehose buffering hints, Tumbling window
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.