No module named 'resource' installing Apache Spark on Windows

Question

I am trying to install apache spark to run locally on my windows machine. I have followed all instructions here https://medium.com/@loldja/installing-apache-spark-pyspark-th

Answer 1

The fix can be found at https://github.com/apache/spark/pull/23055.

The resource module is only for Unix/Linux systems and is not applicaple in a windows environment. This fix is not yet included in the latest release but you can modify the worker.py in your installation as shown in the pull request. The changes to that file can be found at https://github.com/apache/spark/pull/23055/files.

You will have to re-zip the pyspark directory and move it the lib folder in your pyspark installation directory (where you extracted the pre-compiled pyspark according to the tutorial you mentioned)

Answer 2

Adding to all those valuable answers,

For windows users,Make sure you have copied the correct version of the winutils.exe file(for your specific version of Hadoop) to the spark/bin folder

Say, if you have Hadoop 2.7.1, then you should copy the winutils.exe file from the Hadoop 2.7.1/bin folder

Link for that is here

https://github.com/steveloughran/winutils

Answer 3

I struggled the whole morning with the same problem. Your best bet is to downgrade to Spark 2.3.2

Answer 4

I edited worker.py file. Removed all resource-related lines. Actually # set up memory limits block and import resource. The error disappeared.