The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- (on Windows)

Question

I am running Spark on Windows 7. When I use Hive, I see the following error

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions a         


        

Answer 1

This is a simple 4 step process:

For Spark 2.0+:

1. Download Hadoop for Windows / Winutils

2. Add this to your code (before SparkSession initialization):

   if(getOS()=="windows"){
       System.setProperty("hadoop.home.dir", "C:/Users//winutils-master/hadoop-2.7.1"); 
   }   
   

3. Add this to your spark-session (You can change it to C:/Temp instead of Desktop).

   .config("hive.exec.scratchdir","C:/Users//Desktop/tmphive")
   

4. Open cmd.exe and run:

   "path\to\hadoop-2.7.1\bin\winutils.exe" chmod 777 C:\Users\\Desktop\tmphive
   

Answer 2

Error while starting the spark-shell on VM running on Windows: Error msg: The root scratch dir: /tmp/hive on HDFS should be writable. Permission denied

Solution: /tmp/hive is temporary directory. Only temporary files are kept in this location. No problem even if we delete this directory, will be created when required with proper permissions.

Step 1) In hdfs, Remove the /tmp/hive directory ==> "hdfs dfs -rm -r /tmp/hive"

2) At OS level too, delete the dir /tmp/hive ==> rm -rf /tmp/hive

After this, started the spark-shell and it worked fine..

Answer 3

I was getting the same error "The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-" on Windows 7. Here is what I did to fix the issue:

1. I had installed Spark on C:\Program Files (x86)..., it was looking for /tmp/hive under C: i.e., C:\tmp\hive
2. I downloaded WinUtils.exe from https://github.com/steveloughran/winutils. I chose a version same as what I chose for hadoop package when I installed Spark. i.e., hadoop-2.7.1 (You can find the under the bin folder i.e., https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin)
3. Now used the following command to make the c:\tmp\hive folder writable winutils.exe chmod 777 \tmp\hive

Note: With a previous version of winutils too, the chmod command was setting the required permission without error, but spark still complained that the /tmp/hive folder was not writable.

Answer 4

First thing first check your computer domain. Try

c:\work\hadoop-2.2\bin\winutils.exe ls c:/tmp/hive

If this command says access denied or FindFileOwnerAndPermission error (1789): The trust relationship between this workstation and the primary domain failed.

It means your computer domain controller is not reachable , possible reason could be you are not on same VPN as your system domain controller.Connect to VPN and try again.

Now try the solution provided by Viktor or Nishu.

Answer 5

You don't have to fix the permission of /tmp/hive directory yourself (like some of the answers suggested). winutils can do that for you. Download the appropriate version of winutils from https://github.com/steveloughran/winutils and move it to spark's bin directory (e. x. C:\opt\spark\spark-2.2.0-bin-hadoop2.6\bin). That will fix it.

Answer 6

There is a bug in Spark Jira for the same. This has been resolved few days back. Here is the link.

https://issues.apache.org/jira/browse/SPARK-10528

Comments have all options, but no guaranteed solution.