Spark optimization - joins - very low number of task - OOM
问题 My spark application fail with this error : Exit status: 143. Diagnostics: Container killed on request. Exit code is 143 This is what i get when I inspect the containger log : java.lang.OutOfMemoryError: Java heap space My application is mainly get a table then join differents tables that i read from aws S3: var result = readParquet(table1) val table2 = readParquet(table2) result = result.join(table2 , result(primaryKey) === table2(foreignKey)) val table3 = readParquet(table3) result = result