Transforming data in motion

Lesson 3 of 5 in Transforming Data and Engineering Features.

Not every transform can wait for a nightly batch. Two tools transform data in motion, split by one question: does the transform need to remember anything between records? AWS Lambda is the stateless option — Firehose can invoke a Lambda on each buffered batch to filter, reshape, or enrich records in flight (and Firehose separately converts JSON to Parquet/ORC natively, using a schema from the Glue Data Catalog). Managed Flink is the stateful option — windows, joins, and sessionization over a stream, with event-time semantics.

Stateless vs stateful stream transforms
Lambda (in Firehose / on Kinesis)Managed Service for Apache Flink

State between records

None

Managed, checkpointed state

Typical job

Filter bad records, reshape JSON, enrich one record from a lookup

Sliding-window counts, stream-to-stream joins, sessionization

Time handling

Processes what arrives, batch by batch

Event time, watermarks, late-data handling

Fit

Cheap glue for simple per-record work

Real-time feature computation

When Lambda is enough: the transform looks at one record at a time — scrub a field, drop malformed rows, standardize timestamps, call out for a static enrichment. The moment the requirement aggregates across records or across time, Lambda would need a database to fake state; Flink has state as a first-class, checkpointed feature. That line decides the exam question.

Route the event: stateless or stateful?

  1. Event arrives on the stream
  2. Needs memory across records?

    The deciding question: does the transform aggregate across records or across time — windows, joins, sessions — or does it look at one record at a time?

  3. Lambda transform

    Stateless per-record work in Firehose (or on Kinesis): drop malformed rows, reshape JSON, standardize timestamps, enrich from a static lookup.

  4. Managed Flink

    Checkpointed state as a first-class feature: sliding windows, stream-to-stream joins, sessionization — with event-time watermarks for late data.

  5. Cleaned records land in S3

    Firehose buffers the transformed batch and delivers it to the lake.

  6. Feature Store online store

    Rolling features like txn_count_10min land where the endpoint reads fresh values at scoring time.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.