Concurrency, quotas, and ceilings

Lesson 4 of 4 in Automating Data Processing.

At small scale a data pipeline is limited by its own code. At real scale it is limited by numbers somebody else chose, and the first time you meet one it does not look like a limit — it looks like an intermittent failure at 06:00 that nobody can reproduce at 11:00.

The ceilings that decide designs in this domain are few enough to list, and the list is worth knowing cold:

Glue, per job: MaxConcurrentRuns, default 1. Beyond it, StartJobRun returns ConcurrentRunsExceededException.
Glue, per account, per Region: maximum concurrent job runs, and maximum task DPUs — a shared pool, so another team's oversized job is a reason yours is refused.
Athena: active DML queries, which counts running and queued queries; past it you get TooManyRequestsException. Active DDL queries have their own quota.
Lambda: a regional concurrent-execution pool, sliced by reserved concurrency.
Kinesis Data Streams, per shard: 1 MB/s or 1,000 records/s in, 2 MB/s out — and five GetRecords calls per second.

And there are exactly four responses. Back off with jitter. Queue the work. Reserve dedicated capacity. Raise the quota. Most exam questions in this area are asking which of the four a constraint implies — and the giveaway is whether the scenario cares about latency, loss, or predictable cost.

Three of those ceilings have subtleties that are asked about directly.

Athena's active DML quota counts queued queries. This surprises people, because it means a well-behaved client that submits twenty-six queries and waits patiently can still receive TooManyRequestsException on the twenty-sixth — the first twenty-five are active, several of them merely waiting for resources. Raising the quota does not make more queries run; it lets more of them sit in the queued state. If what you actually need is guaranteed concurrency for a critical workload, the answer is an Athena capacity reservation, where reserved capacity is not competing for the shared pool and the bill becomes DPU-hours instead of bytes scanned.

Glue's per-job setting sits underneath two account-level ones. Raising MaxConcurrentRuns from 1 to 50 on one job does nothing if the account's concurrent-run or task-DPU limit is reached first — and worse, it may let one job consume the shared pool and starve everything else. Job queuing is the answer to burstiness: a run that would exceed a limit is enqueued and reattempted as capacity frees, converting failures into latency. Note the operational consequence, because it is the kind of detail a scenario hinges on — a queued run has not started, so an alarm measuring duration from submission will now fire for reasons that are nothing to do with the job.

Concurrency and correctness are different problems. Raising concurrency on a job with bookmarks enabled is not a scaling decision, it is a correctness decision, because two runs can read and advance the same state. The same logic applies to anything that writes to a fixed location: two runs writing the same partition is a race whatever the quota says.

The four responses to a ceiling, and the sentence in a scenario that selects each one. Note that only one of them reduces cost, and only one of them guarantees anything.
Response	What it converts	Costs	Selected by a scenario that says
Back off with jitter	a failure into a later success	latency, and complexity in the caller	"transient throttling", "the SDK retries already" — the default, not a design
Queue (job queuing, SQS)	burstiness into latency, with nothing lost	queue depth, and alarms that must measure the right thing	"a burst every morning", "no submission may be dropped"
Reserve capacity	contention into a guarantee	idle capacity you still pay for	"predictable cost", "this workload must never be queued behind ad-hoc queries"
Raise the quota	a hard stop into a higher hard stop	nothing directly — and nothing is guaranteed either	"we have grown", "a one-off migration" — never a fix for a design that fans out without bound

The fan-out that eats its own quota

The most common self-inflicted ceiling in this domain has the same shape every time.

A per-object trigger fans out one unit of work per file. A nightly upload of 40,000 small files becomes 40,000 invocations, each of which runs a query, or starts a job, or opens a database connection. Nothing is misconfigured; the design simply multiplied the arrival rate by the file count and handed the result to a service with a quota.

Three fixes, in order of preference. Fan in — buffer the events through a queue and process them in batches, or let the arrival merely record that work is pending and have a scheduled job do it once. Bound the fan-out explicitly — Distributed Map with a maximum concurrency, rather than unbounded parallelism. Fix the file count upstream — a Firehose buffering interval, or a compaction pass, turns 40,000 objects into 400 and makes the whole question disappear along with the small-files penalty.

The diagnostic that identifies it: the failure rate scales with the number of input files, not with the volume of data.

Key terms: MaxConcurrentRuns, AWS Glue job queuing, DPU (data processing unit), Throttling ceiling, Exponential backoff with jitter

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.