apache-spark-standalone

What is the relationship between workers, worker instances, and executors?

阅读更多关于 What is the relationship between workers, worker instances, and executors?

问题 In Spark Standalone mode , there are master and worker nodes. Here are few questions: Does 2 worker instance mean one worker node with 2 worker processes? Does every worker instance hold an executor for specific application (which manages storage, task) or one worker node holds one executor? Is there a flow chart explain how spark runtime, such as word count? 回答1: I suggest reading the Spark cluster docs first, but even more so this Cloudera blog post explaining these modes. Your first

Apache Spark: Differences between client and cluster deploy modes

阅读更多关于 Apache Spark: Differences between client and cluster deploy modes

TL;DR: In a Spark Standalone cluster, what are the differences between client and cluster deploy modes? How do I set which mode my application is going to run on? We have a Spark Standalone cluster with three machines, all of them with Spark 1.6.1: A master machine, which also is where our application is run using spark-submit 2 identical worker machines From the Spark Documentation , I read: (...) For standalone clusters, Spark currently supports two deploy modes. In client mode, the driver is launched in the same process as the client that submits the application. In cluster mode, however,

Spark Standalone Number Executors/Cores Control

阅读更多关于 Spark Standalone Number Executors/Cores Control

So I have a spark standalone server with 16 cores and 64GB of RAM. I have both the master and worker running on the server. I don't have dynamic allocation enabled. I am on Spark 2.0 What I dont understand is when I submit my job and specify: --num-executors 2 --executor-cores 2 Only 4 cores should be taken up. Yet when the job is submitted, it takes all 16 cores and spins up 8 executors regardless, bypassing the num-executors parameter. But if I change the executor-cores parameter to 4 it will adjust accordingly and 4 executors will spin up. Disclaimer : I really don't know if --num-executors

Which cluster type should I choose for Spark?

阅读更多关于 Which cluster type should I choose for Spark?

问题 I am new to Apache Spark, and I just learned that Spark supports three types of cluster: Standalone - meaning Spark will manage its own cluster YARN - using Hadoop's YARN resource manager Mesos - Apache's dedicated resource manager project Since I am new to Spark, I think I should try Standalone first. But I wonder which one is the recommended. Say, in the future I need to build a large cluster (hundreds of instances), which cluster type should I go to? 回答1: I think the best to answer that

Spark Standalone Number Executors/Cores Control

阅读更多关于 Spark Standalone Number Executors/Cores Control

问题 So I have a spark standalone server with 16 cores and 64GB of RAM. I have both the master and worker running on the server. I don\'t have dynamic allocation enabled. I am on Spark 2.0 What I dont understand is when I submit my job and specify: --num-executors 2 --executor-cores 2 Only 4 cores should be taken up. Yet when the job is submitted, it takes all 16 cores and spins up 8 executors regardless, bypassing the num-executors parameter. But if I change the executor-cores parameter to 4 it