Spark Exception: Python in worker has different version 3.4 than that in driver 3.5

Question

I am using Amazon EC2, and I have my master and development servers as one. And I have another instance for a single worker.

I am new to this, but I have managed to make

Answer 1

Since you already use Anaconda you can simply create an environment with the desired Python version:

conda create --name foo python=3.4
source activate foo

python --version
## Python 3.4.5 :: Continuum Analytics, Inc

and use it as PYSPARK_DRIVER_PYTHON:

export PYSPARK_DRIVER_PYTHON=/path/to/anaconda/envs/foo/bin/python