Ingestion Patterns

The first decision in any pipeline: streaming or batch, and what each commits you to. Throughput and latency targets, bounded versus unbounded data, replayability, the cost of getting it wrong in both directions, and the AWS service that matches each shape.

Content current as of 2026-10.

Lessons

1. Streaming or batch — the deciding constraints
2. Batch ingestion and transfer
3. Change data capture from databases
4. Throttling, retries, and backpressure

Sources

• Amazon Kinesis Data Streams developer guide — quotas and limits
• AWS Database Migration Service user guide — ongoing replication (CDC)
• Amazon S3 user guide — uploading objects using multipart upload
• AWS SDKs and Tools reference guide — retry behavior
• AWS cloud data ingestion patterns and practices (whitepaper)