How to specify the version of Python for spark-submit to use?

Question

I have two versions of Python. When I launch a spark application using spark-submit, the application uses the default version of Python. But, I want to use the other one. How to

Answer 1

If you want to specify the option PYSPARK_MAJOR_PYTHON_VERSION in spark-submit command line, you should check this:

http://spark.apache.org/docs/latest/running-on-kubernetes.html

You can search spark.kubernetes.pyspark.pythonVersion in this page and you'll find following content:

spark.kubernetes.pyspark.pythonVersion  "2" This sets the major Python version of the docker image used to run the driver and executor containers. Can either be 2 or 3.

Now, your command should looks like :

spark-submit --conf spark.kubernetes.pyspark.pythonVersion=3 ...

It should work.