Spark query running very slow

Question

i have a cluster on AWS with 2 slaves and 1 master. All instances are of type m1.large. I\'m running spark version 1.4. I\'m benchmarking the performance of spark over 4m data c

Answer 1

1. Set default.parallelism to 2
2. Start spark with --num-executor-cores 8
3. Modify this part

df.registerTempTable('test') d=sqlContext.sql("""...

to

df.registerTempTable('test') sqlContext.cacheTable("test") d=sqlContext.sql("""...