Apache Hadoop YARN Resource Management System

A Apache Hadoop YARN Resource Management System is a resource management system that can support apache hadoop yarn resource management tasks through yarn resource negotiation.

• AKA: YARN, Yet Another Resource Negotiator, Hadoop YARN, MapReduce v2 (MRv2), Apache YARN, Hadoop 2.0 Resource Manager.
• Context:
  • It can typically separate YARN Resource Management from yarn job scheduling through yarn architectural separation.
  • It can typically coordinate YARN ResourceManagers with yarn nodemanagers via yarn heartbeat protocols.
  • It can typically negotiate YARN Containers through yarn applicationmasters and yarn resource requests.
  • It can typically implement Policies including yarn capacityschedulers, yarn fairschedulers, and yarn fifoschedulers.
  • It can typically monitor YARN Applications via yarn application timeline servers and yarn resource tracking.
  • It can typically handle YARN Node Failures through yarn fault tolerance mechanisms and yarn container recovery.
  • It can often support YARN Federation for yarn multi-cluster management and yarn resource sharing.
  • It can often enable YARN Label-Based Scheduling through yarn node labels and yarn queue mappings.
  • It can often integrate with YARN-Compatible Frameworks including apache spark, apache flink, and apache storm.
  • It can range from being a Single-Cluster YARN System to being a Federated YARN System, depending on its yarn cluster scope.
  • It can range from being a Homogeneous YARN System to being a Heterogeneous YARN System, depending on its yarn node diversity.
  • It can range from being a Fair-Share YARN System to being a Capacity-Based YARN System, depending on its yarn scheduling policy.
  • It can range from being a Small YARN Cluster to being a Large YARN Cluster, depending on its yarn cluster size.
  • ...
• Examples:
  • YARN ResourceManager Components, such as:
    • YARN Scheduler for yarn resource allocation decisions.
    • YARN ApplicationManager for yarn application lifecycle management.
  • YARN NodeManager Components, such as:
    • YARN Container Executor for yarn container launch.
    • YARN Node Health Checker for yarn node monitoring.
  • Production YARN Deployments, such as:
    • Yahoo! YARN Cluster with 40,000+ nodes (2013-2017).
    • Twitter YARN Infrastructure processing 1TB/sec of data.
    • eBay YARN Platform supporting 10,000+ nodes.
  • YARN MRv2 Framework, Apache Hadoop 2.0 (2013).
  • Apache Hadoop v3 YARN with gpu resource support (2017).
  • ...
• Counter-Examples:
  • Hadoop MapReduce v1, which uses jobtracker-based resource management.
  • Apache Mesos Platform, which provides offer-based resource allocation.
  • Kubernetes System, which focuses on container orchestration.
  • Spark Standalone Manager, which provides spark-specific resource management.
• See: Apache Hadoop Data-Processing Framework, YARN MRv2 Framework, Distributed Resource Management System, Big Data Processing Platform, Cluster Resource Manager, MapReduce Framework, Distributed Resource Control System, Apache Spark Platform, Container Resource Management.

---