Apache Spark YARN mode startup takes too long (10+ secs)

Question

I’m running a spark application with YARN-client or YARN-cluster mode.

But it seems to take too long to startup.

It takes 10+ seconds to initialize the spark con

Answer 1

If you're using mac os to run some tasks in standalone mode,

Just remember to enable the remote ssh connection in your system preference -> sharing (I don't know the reason why we should do that)

Before enabling it, it takes me about 1 minute to run spark-submit xx.py

After enabling it, it only takes me 3 seconds to run it.

I hope it can help others has the kind of issues on Mac os platform