What to do with “WARN TaskSetManager: Stage contains a task of very large size”?

Question

I use spark 1.6.1.

My spark application reads more than 10000 parquet files stored in s3.

val df = sqlContext.read.option(\"mergeSchema\", \"true\").pa         


        

Answer 1

The issue is that your dataset is not evenly distributed across partitions and hence some partitions have more data than others (and so some tasks compute larger results).

By default Spark SQL assumes 200 partitions using spark.sql.shuffle.partitions property (see Other Configuration Options):

spark.sql.shuffle.partitions (default: 200) Configures the number of partitions to use when shuffling data for joins or aggregations.

A solution is to coalesce or repartition your Dataset after you've read parquet files (and before executing an action).

Use explain or web UI to review execution plans.

---

The warning gives you a hint to optimize your query so the more effective result fetch is used (see TaskSetManager).

With the warning TaskScheduler (that runs on the driver) will fetch the result values using the less effective approach IndirectTaskResult (as you can see in the code).