Batch ingestion: S3, Glue, and database extracts
Lesson 2 of 5 in Ingesting and Storing Training Data.
Amazon S3 is the center of gravity for ML on AWS: SageMaker training jobs read their input channels from it, Glue writes to it, Athena queries it, and the Feature Store's offline half lives on it. So batch ingestion is really one question — where does the data live today, and what is the standard path from there into S3? Transactions sit in RDS/Aurora, key-value items in DynamoDB, logs and search indexes in OpenSearch Service, files on EFS, and block volumes (EBS) belong to a single instance.
| Where it sits | What it is | Standard path to training |
|---|---|---|
Amazon S3 | Object storage — the lake itself | Already home. Point Glue, Athena, and SageMaker at it; do not copy it anywhere. |
RDS / Aurora | Relational OLTP tables | AWS Glue over a JDBC connection for scheduled extracts; AWS DMS when you need continuous replication. |
DynamoDB | NoSQL key-value items | Native export to S3 (reads a backup, zero impact on table throughput) or the Glue DynamoDB connector. |
OpenSearch Service | Search / log analytics index | Glue connector or an export job into S3 — indexes are for querying, not for training reads. |
Amazon EFS | Shared NFS file system | SageMaker training can mount EFS directly as a data source; DataSync copies it to S3 for lake workflows. |
Amazon EBS | Block volume on one instance | Instance-local scratch space — not shared. Anything durable gets copied to S3. |
Merging is the second half of batch ingestion. AWS Glue runs serverless Spark: a job joins the RDS extract with clickstream already in S3, deduplicates, and writes partitioned Parquet back to the lake — the canonical nightly picture. (EMR does the same with more control; module 2 draws that line properly.)
The last hop is how a training job reads S3, and the exam cares about the two input modes.
The batch road, end to end
- RDS / Aurora tables
Relational OLTP rows, extracted over a JDBC connection on a schedule.
- DynamoDB table
Key-value items. The native export to S3 reads from a backup, so production throughput never notices.
- Glue crawler
Samples the sources, infers schemas, and registers tables and partitions in the Data Catalog.
- Glue Data Catalog
The metadata spine: the ETL job (and later Athena and SageMaker) resolve tables here.
- Glue ETL job
Serverless Spark joins the extracts with what already sits in the lake, deduplicates, and cleans.
- Partitioned Parquet in S3
Columnar, splittable, pruned by partition — the lake's working format for everything downstream.
- SageMaker training channel
File mode downloads the channel up front; FastFile streams it on first read.
File mode
The default: SageMaker downloads the entire channel to the training instance's volume before training starts. Simple and gives random access — but startup time and required disk grow with the dataset. A multi-terabyte channel can keep instances waiting (and billing) long before the first step.
FastFile mode
Presents S3 objects as if they were local files but streams them on first read. Training starts immediately, needs no volume big enough for the whole dataset, and shines when large data is read sequentially. No code changes — your script still sees file paths.
Pipe mode (the ancestor)
The original streaming mode, pushing records through a named pipe (happiest with RecordIO). FastFile delivers the same streaming benefit with ordinary file semantics, so new work rarely reaches for Pipe — but the name still appears in older material.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.