Spark fails to start in local mode when disconnected [Possible bug in handling IPv6 in Spark??]

Question

The problem is the same as described here Error when starting spark-shell local on Mac

... but I have failed to find a solution. I also used to get the malformed URI err

Answer 1

I am not sure if this will help you, but it solved my problem on Mac.

1) Get your hostname. (In terminal, this is usually the first part of the line (before the @ in Linux, before the : in Mac)) (In Mac, you can also type hostname in terminal to get your hostname)

2) In /etc/hosts add:

127.0.0.1 whatever-your-hostname-is

For me, I originally had

127.0.0.1 localhost

but I changed it to

127.0.0.1 my-hostname

Save this change and retry pyspark.

I got this solution from this stackoverflow: Mac spark-shell Error initializing SparkContext

I hope this helps you.