Ingesting and Storing Training Data

Formats that make or break training, batch extraction from S3 and databases, streaming ingestion with Kinesis and Flink, SageMaker Feature Store, and where vectors live for AI apps.

Content current as of 2026-10.

Lessons

1. File formats that make or break training: Parquet, ORC, CSV, JSON
2. Batch ingestion: S3, Glue, and database extracts
3. Streaming ingestion: Kinesis, Firehose, Flink, Kafka
4. SageMaker Feature Store: online and offline
5. Vector stores for AI apps: OpenSearch, pgvector, S3

Sources

• Amazon Athena — columnar storage formats
• Amazon Kinesis Data Streams developer guide
• Amazon Data Firehose developer guide
• Amazon SageMaker Feature Store developer guide