EMR: when the cluster earns its keep

Lesson 3 of 5 in Transformation Engines.

The Glue-or-EMR question comes up on every form, and the honest axis is control versus overhead, not scale. Both scale. What differs is how much of the stack is yours.

Glue gives you serverless Spark with the Data Catalog, bookmarks, DynamicFrames and a visual editor, and no capacity to manage. EMR gives you the release label — which pins Spark, Hadoop and the rest — the applications installed, the instance types, and every configuration classification you care about.

Five things genuinely point at EMR, and a stem usually names one of them:

  • A framework that is not Spark. Trino for interactive SQL across sources, HBase for a wide-column store, Hive for an existing estate, Flink on the cluster.
  • A specific Spark or Hadoop version, or custom JARs and native libraries installed at cluster level.
  • Deep per-cluster tuning — a configuration classification Glue does not expose.
  • Notebook-driven interactive work at scale, where a persistent cluster beats job-per-run.
  • Cost at sustained volume. A long-running cluster on Reserved or Spot capacity beats per-DPU-second pricing once utilisation is high.

Three deployment forms, and the choice between them is a separate question from the choice of EMR itself. EMR on EC2 is the classic cluster, persistent or transient. EMR Serverless provisions workers per job and releases them after, with pre-initialised capacity if you need fast starts. EMR on EKS runs jobs as pods on your existing Kubernetes cluster — and its whole value is consolidation, so it is right for an organisation that already runs EKS and wrong for one that would have to acquire it.

Glue, EMR, or neither?

Interactive decision tree — outcomes:

  • Amazon Athena — do not start a cluster at all

    The most commonly missed answer. If the work is "aggregate these files and give me the result", serverless SQL over S3 needs no engine to size and bills $5 per TB scanned with a 10 MB minimum per query. Add a CREATE TABLE AS SELECT and it writes a transformed dataset too.

    Where it stops: no incremental bookmarks, limited control over output file layout, and a query timeout that long transformations will meet.

  • AWS Glue ETL

    The default for scheduled or event-driven Spark ETL on AWS. You get the Catalog, bookmarks, DynamicFrames, the visual editor and auto scaling, and you never size a cluster.

    Pair it with Flex execution if nothing downstream has a deadline, and with auto scaling so you specify a maximum rather than a fixed worker count.

  • A Glue Python shell job

    0.0625 or 1 DPU, a single process, no Spark at all. The right vehicle for a script that calls APIs, moves metadata, triggers something or runs a short SQL statement — all of which are badly served by starting a distributed engine.

  • EMR on EC2

    A real cluster, with the release label pinning every framework version and the full configuration surface available. Make it transient (terminate on step completion) unless something genuinely needs to persist, put Spot on task nodes, and use the Glue Data Catalog as the metastore so the tables outlive the cluster.

  • EMR Serverless

    EMR release versions and the Spark-native experience with no instances to choose: capacity is expressed as maximum vCPU and memory, and you pay per worker-second. Pre-initialised capacity keeps warm workers so jobs start in seconds, and you pay for that warmth.

    What you give up against a cluster: no persistent on-cluster service, no SSH onto a node, no cluster-level customisation. Against Glue: no bookmarks, no DynamicFrames, no Glue Studio.

  • EMR on EKS

    The EMR runtime and job API with Kubernetes as the resource manager: a virtual cluster maps to an EKS namespace, jobs become pods, and your data workloads share node groups, autoscaling, capacity reservations and the security model with everything else you run.

    Its value is consolidation, so the signal is an organisation that already runs EKS. A team with no EKS footprint should not acquire one to run Spark.

  • EMR for cost, at sustained high utilisation

    Per-second serverless pricing is excellent for intermittent work and loses to a long-running cluster on Reserved or Spot capacity once utilisation is consistently high. The crossover is a real calculation rather than a slogan, and it needs three inputs: DPU-hours actually consumed today, the equivalent instance hours, and the Spot share you can realistically run on task nodes.

    Do the arithmetic before claiming it, because the answer flips on utilisation and most pipelines are less utilised than their owners believe.

Key terms: Amazon EMR, EMR Serverless, EMR on EKS, EMRFS and HDFS, EMR managed scaling and instance fleets