Write and run pyspark in IntelliJ IDEA

Question

i am trying to work with Pyspark in IntelliJ but i cannot figure out how to correctly install it/setup the project. I can work with Python in IntelliJ and I can use the pyspark

Answer 1

1 problem I encountered was space as in 'Program Files\spark' for environment variables SPARK_HOME and PYTHONPATH (as stated above) so I moved spark binaries to my user directory instead. Thanks to this answer. Also, make sure you installed the packages for the environment. Ensure you see pyspark package in Project Structure -> Platform Settings SDK -> Python SDK (of choice) -> Packages.