Apache Spark Cluster Instance

From GM-RKB
Jump to navigation Jump to search

An Apache Spark Cluster Instance is a computing cluster that is based on an Apache Spark framework.



References

2015

  • http://techblog.netflix.com/2015/03/can-spark-streaming-survive-chaos-monkey.html
    • Component
      Type
      Behaviour on Component Failure
      Resilient
      Driver
      Process
      Client Mode: The entire application is killed
      Cluster Mode with supervise: The Driver is restarted on a different Worker node
      Master
      Process
      Single Master: The entire application is killed
      Multi Master: A STANDBY master is elected ACTIVE
      Worker Process
      Process
      All child processes (executor or driver) are also terminated and a new worker process is launched
      Executor
      Process
      A new executor is launched by the Worker process
      Receiver
      Thread(s)
      Same as Executor as they are long running tasks inside the Executor
      Worker Node
      Node
      Worker, Executor and Driver processes run on Worker nodes and the behavior is same as killing them individually

2012