Automating Data Processing

Making the line run without a human in it: scheduling and event triggers, Lambda inside a data pipeline and the places it does not belong, Glue job bookmarks and incremental processing, concurrency and the quota ceilings you meet at scale, and the automation that handles its own failures.

Content current as of 2026-10.

Lessons

  1. Schedules, events, and triggers
  2. Lambda in a data pipeline
  3. Bookmarks and incremental processing
  4. Concurrency, quotas, and ceilings

Sources