Batch ingestion and transfer
Lesson 2 of 4 in Ingestion Patterns.
Batch ingestion questions are arithmetic questions wearing a service-selection costume. The sentence that decides them is almost always a volume and a deadline, and the first thing to do is divide one by the other.
Keep one number: a saturated 100 Mbps link moves about 1 TB per day. Scale it linearly. A 1 Gbps circuit is roughly 10 TB a day if nothing else uses it, which is never true, so halve your estimate for a shared link. Now a stem that says "480 TB to move, and the project has four weeks" answers itself: that is 17 TB a day, which needs 2 Gbps of dedicated, fully utilised bandwidth for the entire month, and a courier starts to look reasonable rather than exotic.
The second thing to notice is whether the requirement is a backlog or a flow. A backlog is finite and happens once — a data-centre exit, a historical archive. A flow recurs — a nightly delta, a partner feed. The services split cleanly along that line, and offering a flow answer to a backlog question (or the reverse) is the most common way to get one of these wrong.
| Service | Answers the question | The giveaway phrase | Wrong when |
|---|---|---|---|
move or synchronise large volumes between on-premises storage (NFS, SMB, HDFS, object) and AWS storage, on a schedule, with verification | "every night", "keep in sync", "incremental", "verify integrity" | the network cannot carry it in the time available, or the partner is pushing rather than you pulling | |
expose a managed SFTP, FTPS, FTP or AS2 endpoint that writes into Amazon S3 or EFS | "our partner uploads via SFTP", "they will not change their process", "AS2" | you control both ends — then DataSync or a direct SDK upload is simpler and cheaper | |
shift a finite backlog whose network transfer time is unacceptable, or pre-process at a disconnected site | a large volume plus a short deadline, "limited connectivity", "the data cannot traverse the internet" | the requirement is ongoing — a device solves a backlog, never a flow | |
AWS Direct Connect | provide dedicated, predictable private bandwidth for sustained transfer and hybrid access | "consistent network performance", "private connectivity", a recurring large volume | the need is one-time and urgent — provisioning a circuit takes longer than shipping a device |
upload a single large object reliably and in parallel from a client you control | "a 200 GB file", "the upload keeps failing partway", "resume" | the problem is many files or an unreliable site rather than one big object | |
pull from a software-as-a-service application whose API you would otherwise have to write a client for | a named SaaS product — Salesforce, ServiceNow, Zendesk — plus "no custom code" | the source is a database or a file share, or the transformation needed is more than field mapping and filtering |
A line chart with volume in terabytes on a logarithmic x axis from 1 to 1000, and transfer days on a logarithmic y axis. Three lines, each computed as volume divided by daily throughput at full saturation: 100 Mbps moves about 1 TB per day, so the line passes through 1 TB at 1 day and 1000 TB at 1000 days; 1 Gbps moves about 10.8 TB per day, passing through 10 TB at roughly 0.9 days and 1000 TB at roughly 93 days; 10 Gbps moves about 108 TB per day, passing through 100 TB at roughly 0.9 days and 1000 TB at roughly 9 days. A horizontal reference line sits at 7 days, marking a typical one-week migration window. Reading the chart, a one-week window admits about 7 TB over 100 Mbps, 75 TB over 1 Gbps, and 750 TB over 10 Gbps — and those figures assume the link carries nothing else, which is why halving them is the honest planning assumption. Any requirement that falls above the 7-day line for the bandwidth actually available is a requirement for a physical transfer device rather than a faster script.
Why does a single large gzip file make my batch job slow rather than fast?
Because gzip is not splittable. Spark, Athena and Hive parallelise by handing each task a byte range, and a reader cannot begin decoding a DEFLATE stream at an arbitrary offset — so one 20 GB .csv.gz object is read by exactly one task, on one core, however large the cluster. The job is not broken; it is serial, and the Spark UI shows one task running for hours while everything else idles.
Three ways out, in order of preference. Ask the producer for many moderate files instead of one enormous one, because the unit of parallelism then becomes the file and 200 x 100 MB gzip objects parallelise perfectly. Convert to Parquet with Snappy, where the container splits at row-group boundaries even though each block is compressed. Or use bzip2 if a single large text file genuinely must stay text.
The partner sends a full snapshot every night, not a delta. Is that a problem?
It is a cost problem and a correctness opportunity, in that order.
The cost: you are moving and storing the whole table daily. If it is 200 GB and 0.5% changes, 99.5% of the transfer is waste. DataSync will not help — the file is genuinely different each night, so there is no delta to detect at the file level.
The opportunity: a full snapshot lets you compute the delta yourself reliably, by comparing today's snapshot to yesterday's on the business key. That catches deletes, which a partner's own "changed rows" extract usually does not — a row that vanished from the source rarely appears in a delta feed. So a nightly snapshot is often more trustworthy than a nightly delta, and the right design keeps the last two snapshots and derives the change set, writing it with a merge rather than an append.
When is a one-time Snow device genuinely wrong even though the arithmetic favours it?
When the data is still changing. A device is a snapshot in transit: whatever is written to it is as of the moment it was written, and by the time it is unloaded in an AWS Region the source has moved on. That is fine for an archive and wrong for a live database.
The standard pattern for that case is a hybrid: ship the historical bulk on a device, and run change data capture over the network for the delta accumulated since the snapshot — which is exactly why DMS offers a full-load-plus-CDC task type. The network only has to carry the changes, which is a tiny fraction of the volume, so a link that could never have moved the whole dataset is entirely adequate.
Key terms: AWS Transfer Family, AWS Snow Family, Multipart upload, Amazon AppFlow, Splittable compression