I\'m new to hadoop and is in learning phase. As per Hadoop Definitve guide, i have set up my hadoop in pseudo distributed mode and everything was working fine. I was even able t
here is the kicker:
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /tmp/hadoop-anshu/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
i'd been having similar issues. i used stop-all.sh to shut down hadoop. i guess it was foolish of me to think this would properly save the data in my HDFS.
but as far as i can tell from what appears to be the appropriate code chunk in the hadoop-daemon.sh script, this is not the case - it just kills the processes:
(stop)
if [ -f $pid ]; then
if kill -0 `cat $pid` > /dev/null 2>&1; then
echo stopping $command
kill `cat $pid`
else
echo no $command to stop
fi
else
echo no $command to stop
fi
did you look to see if the directory it's complaining about existed? i checked and mine did not, although there was an (empty!) data folder in there here I imagine data might have once lived.
so my guess was that what we need to do is configure Hadoop such that our namenode and datanode are NOT stored in a tmp directory. there is some possibility that the OS is doing maintenance and getting rid of these files. either that hadoop figures you don't care about them anymore because you wouldn't have left them in a tmp directory if you did, and you wouldn't be restarting your machine in the middle of a map-reduce job. I don't really think this should happen (i mean, that's not how i would design things) but it seemed like a good guess.
so, based on this site http://wiki.datameer.com/display/DAS11/Hadoop+configuration+file+templates i edited my conf/hdfs-site.xml file to point to the following paths (obviously, make your own directories as you see fit):
<property>
<name>dfs.name.dir</name>
<value>/hadoopstorage/name/</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/hadoopstorage/data/</value>
</property>
Did this, formatted the new namenode (sadly, data loss seems inevitable in this situation), stopped and started hadoop with the shell scripts, restarted the machine, and my files were still there...
YMMV...hope this works for you! i'm on OS X but i don't think you should have dissimilar results.
J
JSH answer is correct.
Just a couple of changes for hadoop 2.6 i had to do:
<property>
<name>dfs.namenode.name.dir</name>
<value>/hadoopstorage/name/</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/hadoopstorage/data/</value>
</property>
See your coresite.xml
in hadoop config directory
vi core-site.xml,hdf.site.xml
Setting this properties in conf/hdfs-site.xml file worked for me!!!
Thanks jsh
<property>
<name>dfs.name.dir</name>
<value>/hadoopstorage/name/</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/hadoopstorage/data/</value>`enter code here`
</property>
Dont forget to set proper permissions to those directories
If you have not resolved the problem, try this:
give the dfs.name.dir
directory to in the user group hadoop
and give the group to write permission.
If you dont care about losing data just execute the command:
./hadoop namenode -format