java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. spark Eclipse on windows 7

Question

I\'m not able to run a simple spark job in Scala IDE (Maven spark project) installed on Windows 7

Spark core dependency has be

Answer 1

1. Download winutils.exe
2. Create folder, say C:\winutils\bin
3. Copy winutils.exe inside C:\winutils\bin
4. Set environment variable HADOOP_HOME to C:\winutils

Answer 2

Follow this:

1. Create a bin folder in any directory(to be used in step 3).

2. Download winutils.exe and place it in the bin directory.

3. Now add System.setProperty("hadoop.home.dir", "PATH/TO/THE/DIR"); in your code.

Answer 3

if we see below issue

ERROR Shell: Failed to locate the winutils binary in the hadoop binary path

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

then do following steps

1. download winutils.exe from http://public-repo-1.hortonworks.com/hdp- win-alpha/winutils.exe.
2. and keep this under bin folder of any folder you created for.e.g. C:\Hadoop\bin
3. and in program add following line before creating SparkContext or SparkConf System.setProperty("hadoop.home.dir", "C:\Hadoop");

Answer 4

I got the same problem while running unit tests. I found this workaround solution:

The following workaround allows to get rid of this message:

    File workaround = new File(".");
    System.getProperties().put("hadoop.home.dir", workaround.getAbsolutePath());
    new File("./bin").mkdirs();
    new File("./bin/winutils.exe").createNewFile();

from: https://issues.cloudera.org/browse/DISTRO-544

Answer 5

On top of mentioning your environment variable for HADOOP_HOME in windows as C:\winutils, you also need to make sure you are the administrator of the machine. If not and adding environment variables prompts you for admin credentials (even under USER variables) then these variables will be applicable once you start your command prompt as administrator.

Answer 6

That's a tricky one... Your storage letter must be capical. For example "C:\..."