Spark: Trying to run spark-shell, but get 'cmd' is not recognized as an internal or

Question

I\'m trying to install Spark on my Windows desktop. Everything should work fine, but I get an error \"\'cmd\' is not recognized as an internal or external command... \"

Answer 1

I was the getting the same error while executing Spark-shell in the command prompt.

I tried everything mentioned above but not able to resolve the issue.

So, at last I added "C:\Windows\System32" in 'PATH' variable of System Variable and it worked.

Answer 2

Check values in JAVA_HOME and make sure it is pointing to correct value. Add %JAVA_HOME%/bin in path value. After modification close command prompt and restart it. Write spark-shell and it will run.

Answer 3

I had the similar error. I fixed it after following changes:

1. There were multiple Java/bin path in the System Path. So I corrected them to reflect single Java/Bin, which is in sync with JAVA_HOME
2. Added C:Windows\system32 to System Path Variable.
3. My Java_Home and java.exe was pointing different places. I fixed them.

Now it works.

Thanks guys.

Answer 4

These are detailed steps to resolve all these issues in Windows

Spark binaries

1. It needs to be in a folder which doesn’t have space, if it must be in a folder which has space then you can use something like Unix’s symlink. I have kept the Spark binaries inside c:\ directly. Set SPARK_HOME to e.g. C:\SparkMedia\spark-2.4.2-bin-hadoop2.7. Please note “bin” is not part of the path.
2. Set “path” for the user and system to %SPARK_HOME%\bin If you decide to keep it outside “program files” (as per point a), then please give permission to the user because Spark keeps temporary files here. If you want to keep Spark inside a directory where user doesn’t have access then you need to put a file %SPARK_HOME%\conf\spark-env.sh with an entry inside it as “SPARK_LOG_DIR=a directory where user has access”.

Winutils

1. It can be inside a folder with space like C:\Program Files (x86)\Winutils Your winutils kept at C:\Program Files (x86)\Winutils works fine.
2. Set HADOOP_HOME to C:\Program Files (x86)\Winutils. Please note “bin” is not part of the path.
3. Set “path” for the user and system to %HADOOP_HOME%\bin

Java

4. It can be inside a folder with space like C:\Program Files\AdoptOpenJDK\jdk-8.0.262.10-hotspot Set JAVA_HOME to C:\Program Files\AdoptOpenJDK\jdk-8.0.262.10-hotspot. Please note “bin” is not part of the path.

5. Set “path” for the user and system to %JAVA_HOME%\bin

With the above setup a user should be able to just type “spark-shell” and spark shell should start.

Answer 5

(I'm not Windows Spark user) The spark-shell.cmd for Windows source code expects "cmd" command is available in PATH.

https://github.com/apache/spark/blob/master/bin/spark-shell.cmd

Would you try adding the directory that contains "cmd.exe" in PATH environment variable? The directory location is shown title bar in your screenshot, and environment variable setting can be done via control panel.

Answer 6

I had the same issue (launch spark-shell and get the the system cannot find the path) After following the above process (changing the spark_home to exclude the /bin it worked fine thanks for sharing guys