Spark java.lang.OutOfMemoryError: Java heap space

Question

My cluster: 1 master, 11 slaves, each node has 6 GB memory.

My settings:

spark.executor.memory=4g, Dspark.akka.frameSize=512

Answer 1

I have few suggession for the above mentioned error.

● Check executor memory assigned as an executor might have to deal with partitions requiring more memory than what is assigned.

● Try to see if more shuffles are live as shuffles are expensive operations since they involve disk I/O, data serialization, and network I/O

● Use Broadcast Joins

● Avoid using groupByKeys and try to replace with ReduceByKey

● Avoid using huge Java Objects wherever shuffling happens

Answer 2

Did you dump your master gc log? So I met similar issue and I found SPARK_DRIVER_MEMORY only set the Xmx heap. The initial heap size remains 1G and the heap size never scale up to the Xmx heap.

Passing "--conf "spark.driver.extraJavaOptions=-Xms20g" resolves my issue.

ps aux | grep java and the you'll see the follow log:=

24501 30.7 1.7 41782944 2318184 pts/0 Sl+ 18:49 0:33 /usr/java/latest/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx30g -Xms20g

Answer 3

Have a look at the start up scripts a Java heap size is set there, it looks like you're not setting this before running Spark worker.

# Set SPARK_MEM if it isn't already set since we also use it for this process
SPARK_MEM=${SPARK_MEM:-512m}
export SPARK_MEM

# Set JAVA_OPTS to be able to load native libraries and to set heap size
JAVA_OPTS="$OUR_JAVA_OPTS"
JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$SPARK_LIBRARY_PATH"
JAVA_OPTS="$JAVA_OPTS -Xms$SPARK_MEM -Xmx$SPARK_MEM"

You can find the documentation to deploy scripts here.

Answer 4

Broadly speaking, spark Executor JVM memory can be divided into two parts. Spark memory and User memory. This is controlled by property spark.memory.fraction - the value is between 0 and 1. When working with images or doing memory intensive processing in spark applications, consider decreasing the spark.memory.fraction. This will make more memory available to your application work. Spark can spill, so it will still work with less memory share.

The second part of the problem is division of work. If possible, partition your data into smaller chunks. Smaller data possibly needs less memory. But if that is not possible, you are sacrifice compute for memory. Typically a single executor will be running multiple cores. Total memory of executors must be enough to handle memory requirements of all concurrent tasks. If increasing executor memory is not a option, you can decrease the cores per executor so that each task gets more memory to work with. Test with 1 core executors which have largest possible memory you can give and then keep increasing cores until you find the best core count.

Answer 5

To add a use case to this that is often not discussed, I will pose a solution when submitting a Spark application via spark-submit in local mode.

According to the gitbook Mastering Apache Spark by Jacek Laskowski:

You can run Spark in local mode. In this non-distributed single-JVM deployment mode, Spark spawns all the execution components - driver, executor, backend, and master - in the same JVM. This is the only mode where a driver is used for execution.

Thus, if you are experiencing OOM errors with the heap, it suffices to adjust the driver-memory rather than the executor-memory.

Here is an example:

spark-1.6.1/bin/spark-submit
  --class "MyClass"
  --driver-memory 12g
  --master local[*] 
  target/scala-2.10/simple-project_2.10-1.0.jar 

Answer 6

Setting these exact configurations helped resolving the issue.

spark-submit --conf spark.yarn.maxAppAttempts=2 --executor-memory 10g --num-executors 50 --driver-memory 12g