Apache Spark architecture

Question

Trying to find a complete documentation about an internal architecture of Apache Spark, but have no results there.

For example I\'m trying to understand next thing: Assu

Answer 1

Here are the answers to your questions

1. Spark will try to load 128Mb chunk into memory and process it in RAM. Keep in mind that that the size in memory can be several times larger than the original size of the raw file due to Java overhead (Java headers, etc). From my experience, it can be 2-4 time larger. If there is not enough memory (RAM) Spark will spill the data to local disk. You may want to tweak these two parameters to minimize the spill: spark.shuffle.memoryFraction and spark.storage.memoryFraction.

2. Spark will always try to use local data from in your HDFS. If the chunk not available locally it will retrieve it from another node in the cluster. more info

3. On shuffle, you do not need to manually save intermediate results to HDFS. Spark will write the results to local storage and shuffle only the data needed maximizing reuse of local storage for the next stage.

Here is good video that goes into more detail of Spark architecture, what happens during shuffle and tips for optimizing performance.