Why Hadoop still matters on AWS

“We use Spark” does not mean Hadoop disappeared. Spark on EMR still schedules containers through YARN (or runs in client/cluster mode against the same cluster), reads through Hadoop-compatible filesystem APIs (s3a://, hdfs://), and often depends on the Hive Metastore for table definitions. MapReduce is rarely written greenfield today, but its execution model—map shards, shuffle, reduce merge—is the mental model behind Spark stages, Hive Tez, and many SQL engines.

EMR’s value is operational: you get versioned EMR releases (e.g. emr-7.x) bundling tested combinations of Hadoop 3.x, Spark, Hive, Livy, Flink, and connectors—plus integration with IAM, CloudWatch, Auto Scaling, Spot, and Step Functions. For pipeline context and Academy framing, see Data Engineering on AWS; for where objects live, see Amazon S3 in depth.

The Hadoop stack: three layers

Think of Hadoop as three cooperating layers:

1. Storage — HDFS or, on modern AWS designs, Amazon S3 via the Hadoop FileSystem abstraction (s3a).
2. Resource management — YARN allocates CPU and memory across applications.
3. Processing frameworks — MapReduce (legacy), Spark, Tez, Hive execution engines, etc.

Common supporting services sit beside this core: ZooKeeper (coordination), the Hive Metastore (catalog), Ranger or IAM-based auth (policy), and optional HBase (wide-column store on HDFS).

HDFS: how distributed storage works

Hadoop Distributed File System (HDFS) splits files into fixed-size blocks (often 128 MiB or 256 MiB) and replicates each block across DataNodes for fault tolerance. The default replication factor is 3: one replica on the local rack, two on other racks when topology is configured.

NameNode and DataNodes

• NameNode — Holds metadata (namespace tree, block locations). It does not store file bytes. A standby NameNode (HA pair) takes over on failure when HA is enabled.
• DataNode — Stores blocks and serves read/write pipelines. Reports heartbeat and block reports to the NameNode.
• Secondary NameNode / Checkpointing — Not a hot standby; assists with fsimage edits checkpointing in non-HA legacy layouts. With HA, JournalNodes persist edits.

Write path: client asks NameNode for block locations → writes to a pipeline of DataNodes → replicas acknowledged. Read path: client gets block locations → reads directly from nearest DataNode.

HDFS on EMR vs S3-first architectures

EMR still runs HDFS on core nodes for scratch, spill, and libraries, but production data lakes almost always use S3 as the system of record. Reasons are operational: S3 decouples storage from compute, survives cluster termination, and bills per GB-month without sizing clusters for disk. HDFS on a small core fleet is fine for temporary shuffle-heavy work; relying on HDFS for durable multi-terabyte datasets ties your data lifecycle to cluster uptime and core-node disk sizing.

The s3a:// filesystem implements Hadoop’s API over S3. Important knobs for engineers: consistent listing (S3 is strongly consistent for new objects today, but LIST at scale still needs prefix discipline), multipart uploads for large writes, server-side encryption (SSE-S3, SSE-KMS), and committers for Spark/Hive so jobs do not leave partial outputs on failure (S3Guard is legacy; modern Spark uses staging/commit protocols appropriate to the EMR release).

YARN: the cluster operating system

Yet Another Resource Negotiator (YARN) separates cluster resource management from application logic. MapReduce v2, Spark on YARN, Hive-on-Tez, and Flink (YARN session) all request containers from YARN.

Key daemons

• ResourceManager (RM) — Global scheduler; tracks cluster capacity and application queues.
• NodeManager (NM) — Per-node agent; launches containers, monitors local resources, reports to RM.
• ApplicationMaster (AM) — One per application; negotiates containers with RM and tracks task progress (Spark’s driver in cluster mode participates in this story).

Queues, fairness, and capacity

Multi-tenant clusters use YARN queues (CapacityScheduler or FairScheduler) with weights, ACLs, and preemption. On EMR, you configure capacity scheduler settings in yarn-site.xml classifications or via the console when creating the cluster. Symptoms of bad queue design: one team’s overnight Spark job starves another’s SLA-bound ETL, or small jobs wait behind a monopolizing “mega-app.”

Sizing heuristic: leave headroom on NodeManagers for HDFS DataNode and OS overhead on core nodes; on task-only nodes you can pack executors tighter. vcores and memory requested by Spark (spark.executor.memory, spark.executor.cores, spark.dynamicAllocation) must fit inside YARN container limits—misalignment produces apps stuck in ACCEPTED or containers OOM-killed.

MapReduce: the original execution engine

MapReduce processes data in two phases:

1. Map — Read input splits (HDFS block-aligned or S3 byte ranges); emit key-value pairs per record.
2. Shuffle & sort — Partition by key, sort locally, transfer across network to reducers.
3. Reduce — Merge sorted runs; compute aggregates or joins per key.

Everything expensive in MapReduce is usually the shuffle: disk spill, network fan-in, and skewed keys (“hot” reducers). Spark improves this with in-memory pipelines and more flexible DAGs, but stage shuffle remains the same bottleneck class—watch spark.sql.shuffle.partitions and skew hints.

You may still see MapReduce in older Hive (MR engine), DistCp-style tooling, or legacy JARs. EMR does not forbid it; it is simply not where new feature work goes.

EMR cluster anatomy

When you launch an EMR cluster, AWS provisions EC2 instances with an EMR AMI and starts services in a defined order (Hadoop → YARN → installed applications).

Instance groups (classic model)

Role	Purpose	Notes
Master	NameNode (if HDFS), ResourceManager, cluster apps (HiveServer2, Spark history server, Zeppelin, etc.)	Typically one instance (or two for HA in advanced setups). Not a place to run heavy executors.
Core	DataNode + NodeManager	Runs tasks and stores HDFS blocks. Scaling down core nodes requires HDFS rebalance/decommission discipline.
Task	NodeManager only	Elastic workers; no HDFS—safe to scale with Spot and terminate when idle.

Instance fleets (flexible model)

Instance fleets let you mix instance types and purchase options (On-Demand + Spot) per role with target capacities and timeouts. AWS places instances from declared fleets—useful for Spot diversification and cost optimization without locking to a single instance type.

Transient vs long-lived clusters

• Transient — Boot → run steps → auto-terminate. Ideal for batch ETL with predictable inputs; you pay only for runtime.
• Long-lived — Shared analytics cluster or notebook hub. Simpler for ad hoc SQL but needs patching discipline, queue governance, and stricter security reviews.

EMR on EKS and EMR Serverless shift operational models: EKS runs Spark jobs on Kubernetes without a classic EC2 cluster; Serverless runs Spark without managing instances at all—pay per vCPU/memory-hour of job execution. Choose them when team skills and workload shape match; classic EMR remains common for Hadoop-native apps and mixed-framework clusters.

EMR releases and applications

An EMR release is a manifest: Hadoop version, Spark, Hive, Pig, HBase, Flink, Livy, Zeppelin, and AWS connectors (for example the EMRFS S3 optimizations). Pin releases in production; upgrading in place can change serializer defaults, Hive metastore compatibility, or Python/Scala versions.

When creating a cluster you select applications to install. Common combinations:

• Spark — Default for batch/stream processing, MLlib, structured streaming.
• Hive — SQL over tables; metastore often externalized to RDS (Apache Hive Metastore) or AWS Glue Data Catalog.
• Tez — Execution engine for Hive instead of MapReduce.
• Hue / Zeppelin — Notebooks and SQL UI (lock down auth in production).
• Presto/Trino — Interactive SQL (often on separate clusters for blast-radius isolation).
• HBase — Low-latency wide-column on HDFS; ops-heavy, co-locate masters thoughtfully.

Spark on EMR (what most teams actually run)

Apache Spark generalizes MapReduce with a DAG of transformations and actions, keeping datasets in memory when possible. On EMR:

• Deploy modes — cluster (driver inside YARN) vs client (driver on edge node or laptop; cluster handles executors).
• Dynamic allocation — Scale executors with load; pair with YARN queue limits.
• Shuffle service — External shuffle service on NodeManagers helps resilience when executors disappear.
• History server — Debug failed stages; logs land in S3 when configured.

Read Parquet/ORC in S3 with partition pruning (spark.sql.sources.partitionColumnTypeInference, metastore stats). For columnar formats and compression in lake design, align with analytics in SQL and relational modeling thinking—schemas still matter at petabyte scale.

Hive, catalog, and table formats

Apache Hive provides SQL and metastore tables over files in S3/HDFS. Modern lakes add:

• Apache Iceberg / Delta Lake / Hudi — Table formats with ACID-ish commits, time travel, and schema evolution on S3.
• AWS Glue Data Catalog — Serverless metastore compatible with Athena, EMR, and Redshift Spectrum.

HiveServer2 on the master exposes JDBC/ODBC; throttle concurrent users on smaller masters. Compile plans to Tez or Spark rather than MapReduce for performance.

Supporting components you will touch in production

• ZooKeeper — Coordination for HBase, Kafka old versions, some HA services; small ensemble, keep off overloaded masters.
• Apache Livy — REST API for Spark sessions (notebooks, job submissions).
• Apache Flink — True streaming with state; YARN per-job or session modes on EMR.
• Sqoop / DistCp — Bulk relational import and distributed copy (still seen in migration playbooks).
• EMRFS — Consistent S3 view and optimizations layered on Hadoop S3A (follow EMR release docs for current recommendations).

Security and governance

• IAM roles for EC2 — Instance profile grants S3, KMS, Glue, and other API access—no long-lived keys on nodes.
• Security configurations — EMR object encapsulating encryption at rest (EBS, S3, in-transit TLS), Kerberos options, and authorization.
• Kerberos — Strong identity for multi-tenant Hadoop; operational overhead (KDC, keytabs, renewals). Common in enterprise hybrid migrations.
• Lake Formation — Fine-grained S3 table/column permissions feeding Athena/EMR through the Glue catalog.
• Network — Clusters in private subnets; restrict security groups; use VPC endpoints for S3/KMS to avoid NAT costs—see AWS network architecture.

EMR blocks public SSH by default in modern wizards; use Systems Manager Session Manager or bastion patterns with least privilege.

Operations: bootstrap actions, steps, and scaling

Bootstrap actions — Scripts run on node provision (install Python deps, agents, tuning). Keep them idempotent and fast; failed bootstrap wastes cluster minutes.

Steps — Ordered work units (Spark jar, Hive script, Pig, shell). A cluster can auto-terminate when steps complete. Use Step concurrency carefully when dependencies exist.

Managed scaling — EMR adjusts task (and sometimes core) capacity based on YARN metrics. Pair with Spot task fleets for cost; cap maximum capacity to prevent runaway bills.

Logs — Push container and step logs to S3 (s3://my-logs/emr/); wire CloudWatch alarms on step failure events or custom metrics.

Cost and performance levers

• Right-size masters — Oversized masters rarely speed up Spark; undersized masters break HiveServer2 and NameNode heap.
• Spot task nodes — Excellent for fault-tolerant batch; avoid Spot-only for stateful HBase masters.
• Graviton instances — Often better price/performance when your AMIs and native deps support ARM.
• Partitioning & file sizing — Millions of tiny objects hurt S3 LIST and job planning; target hundreds of MB per file for analytics.
• Columnar formats — Parquet with Snappy/Zstd compression reduces scan cost versus raw JSON/CSV.
• Transient clusters — Eliminate idle EC2 on nights and weekends.

For FinOps culture around cloud spend—not EMR-specific math—see the FinOps: stop guessing what the cloud costs essay in this blog series.

Troubleshooting playbook

Symptom	Likely layer	What to check
App stuck in ACCEPTED	YARN	Queue capacity, max resources, ACLs; RM UI or yarn application -list
Executor OOM / container killed	Spark + YARN	Memory overhead, partition skew, shuffle size; increase memory or fix hot keys
S3 403 / AccessDenied	IAM	Instance profile policy, bucket policy, KMS key policy, VPC endpoint policy
Slow commits / duplicate outputs	S3 committer	EMR/Spark committer settings for your release; avoid deprecated patterns
HDFS under-replicated blocks	HDFS	Lost core nodes; decommission before terminate; rebalance
Hive metadata errors	Metastore	Glue/RDS connectivity, schema mismatch after upgrade

Use the Spark History Server and YARN ResourceManager UI (with SSH tunnel or EMR Studio) for stage-level timelines. For platform debugging habits on Linux hosts, see Linux in depth.

EMR vs Glue vs Athena (when to use what)

• Athena — Serverless SQL on S3 via Presto/Trino engine; best for ad hoc queries and light ETL without managing clusters.
• Glue — Serverless Spark jobs, crawlers, and scheduling; good for steady ETL with minimal cluster ops.
• EMR — Full control of frameworks, versions, and custom JARs; heavy Spark/Hive/Flink/HBase; hybrid with on-prem Hadoop skills.

Many architectures use all three: Glue catalog + Athena for exploration + EMR for heavy transform or ML feature engineering.

Reference architecture: S3 data lake batch pipeline

Ingest (Kinesis Firehose / DMS / app uploads)
  → s3://data-lake/raw/…
EMR transient cluster (Spark + Glue Catalog)
  → validate, dedupe, partition (event_date=…)
  → s3://data-lake/curated/… (Parquet)
Athena / Redshift Spectrum / OpenSearch export
  → dashboards, ML training, operational metrics

Run the EMR cluster only for the transform window; store scripts in Git; parameterize dates via Step Arguments; emit CloudWatch metrics on records written and data quality checks failed.

Architect checklist

• Pin EMR release and document application versions in runbooks.
• Store durable data in S3 with encryption and lifecycle rules.
• Use task fleets + Spot for elastic compute; keep masters stable.
• Externalize metastore (Glue or RDS), not derby on master.
• Configure logs to S3 and alarms on failed steps.
• Define YARN queues per team/SLA before sharing a long-lived cluster.
• Test failure modes—Spot loss, S3 throttling, skewed joins—before production SLAs.

Further reading

• AWS documentation — Amazon EMR Management Guide, EMR release versions, best practices for Spark tuning
• Apache Hadoop — HDFS architecture, YARN capacity scheduler
• Apache Spark — tuning guide, structured streaming, shuffle behavior
• Delta Lake / Apache Iceberg — table format specs for S3 lakes

Blog index · Data Engineering · Amazon S3 in depth · EBS, S3, and EFS · SQL course

Back to blog list