Apache Spark YARN mode startup takes too long (10+ secs)

Question

I’m running a spark application with YARN-client or YARN-cluster mode.

But it seems to take too long to startup.

It takes 10+ seconds to initialize the spark con

Answer 1

You could check Apache Livy which is a REST API in front of Spark.

• http://livy.io/
• https://github.com/cloudera/livy

You could have one session and multiple requests to that one Spark/Livy session.