Lambda in a data pipeline

Lesson 2 of 4 in Automating Data Processing.

Lambda is excellent at four jobs in a data pipeline and bad at a fifth, and the fifth is the one people reach for.

It is excellent as glue between events and services — an object landed, so start a workflow, tag the object, write a catalog entry. Excellent as a per-record transformer on a stream, through Firehose's transformation hook or an event source mapping on a stream or queue. Excellent as a small-batch processor, where "small" means something that fits comfortably in one invocation's memory and finishes well inside fifteen minutes. And excellent as an API or control-plane caller — the step in a workflow that calls three services and decides something.

It is bad as an ETL engine. Not because it cannot technically do it, but because the limits are hard and they arrive without warning: a maximum timeout of 15 minutes, memory from 128 MB to 10,240 MB, ephemeral /tmp from 512 MB to 10,240 MB, a 6 MB request and response payload on a synchronous invoke, and no shuffle, no spill, no cluster. A function that processes a 200 MB file today processes a 2 GB file in six months by timing out at minute fifteen, having written a third of its output. Spark exists because that failure mode is unacceptable.

The useful mental test is not "is this data big" — it is "what happens to this code when the input is ten times larger?" If the answer is "it runs ten times longer", Lambda has a cliff and Glue does not. If the answer is "there are ten times as many invocations", Lambda is exactly right, because that is the dimension it scales in.

Two more things about Lambda in a pipeline are worth stating precisely, because both are asked as scenarios rather than as facts.

Concurrency is a downstream problem, not a Lambda problem. The regional pool defaults to 1,000 concurrent executions, and a burst of S3 events will happily use all of it — which is marvellous until the target is a relational database with two hundred connections, at which point your pipeline is a denial-of-service attack on your own warehouse. Reserved concurrency is the fix, and it does two things from one number: it guarantees the function that much and it caps the function at that much. Set it to what the downstream can survive and let the queue hold the rest. Setting it to zero throttles the function entirely, which is the documented emergency stop for a runaway consumer without deleting its trigger.

Firehose transformation is its own shape, with its own contract. A Firehose delivery stream can invoke a Lambda function on each buffered batch, and your function must return every record it was given with a result of Ok, Dropped, or ProcessingFailed — returning fewer records than you received is an error, not a filter. Records you mark as failed go to an error prefix in the destination bucket rather than vanishing. This is the right place for per-record parsing, enrichment and PII redaction on the way into the lake, and the wrong place for anything that needs to see more than one batch, because batches are exactly as independent as they look.

Key terms: Lambda event source mapping, Reserved concurrency, Dead-letter queue, Distributed Map

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.