Spark ignores SPARK_WORKER_MEMORY?

Question

I\'m using standalone cluster mode, 1.5.2.

Even though I\'m setting SPARK_WORKER_MEMORY in spark-env.sh, it looks like this setting is igno

Answer 1

You are using wrong setting in cluster mode.

SPARK_EXECUTOR_MEMORY is the right option to set Executor memory in cluster mode.

SPARK_WORKER_MEMORY works only in standalone deploy mode.

Otherway to set executor memory from command line : -Dspark.executor.memory=2g

Have a loook at one more related SE question regarding these settings :

Spark configuration, what is the difference of SPARK_DRIVER_MEMORY, SPARK_EXECUTOR_MEMORY, and SPARK_WORKER_MEMORY?

Answer 2

When using spark-shell or spark-submit, use the --executor-memory option.

When configuring it for a standalone jar, set the system property programmatically before creating the spark context.

System.setProperty("spark.executor.memory", executorMemory)

Answer 3

I've encountered the same problem as yours. The reason is that, in standalone mode, spark.executor.memory is actually ignored. What has an effect is spark.driver.memory, because the executor is living in the driver.

So what you can do is to set spark.driver.memory as high as you want.

This is where I've found the explanation: How to set Apache Spark Executor memory

Answer 4

This is my configuration on cluster mode, on spark-default.conf

spark.driver.memory 5g
spark.executor.memory   6g
spark.executor.cores    4

Did have something like this?

If you don't add this code (with your options) Spark executor will get 1gb of Ram as default.

Otherwise you can add these options on ./spark-submit like this :

# Run on a YARN cluster
export HADOOP_CONF_DIR=XXX
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode cluster \  # can be client for client mode
  --executor-memory 20G \
  --num-executors 50 \
  /path/to/examples.jar \
  1000

Try to check on master(ip/name of master):8080 when you run an application if resources have been allocated correctly.