Problem starting tasktracker in hadoop under windows

Question

I am trying to use hadoop under windows and I am running into a problem when I want to start tasktracker. For example:

$bin/start-all.sh

th

Answer 1

I was running into this issue on an installation of 1.0.3 on Windows server. I changed the default directory in hdfs-site.xml so that the directory that hadoop creates for the dfs is a subdir of the cygwin directory like this...

...

 <property>
    <name>dfs.name.dir</name>
    <value>c:/cygwin/usr/mydir/dfs/logs</value>
 </property>
 <property>
    <name>dfs.data.dir</name>
    <value>c:/cygwin/usr/mydir/dfs/data</value>
 </property>
</configuration>

This seemed to resolve the problem.

The apache documentation for the config files is here

Answer 2

This issue was raised on the Apache Hadoop user mailing list. It appears to be a problem in some release versions of Hadoop and not others.

A simple solution is to download a different version of Hadoop (assuming you do not require a specific Hadoop version for some other reason).

I encountered this exact issue with version 1.0.0 (beta).

I then tried 0.23.0 but got a fatal ClassNotFoundException:

log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named "NullAppender".
Exception in thread "main" java.lang.ClassNotFoundException: hadoop-mapreduce-examples-0.23.0.jar
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:182)

Finally I tried version 0.22.0 and that worked without error. Therefore I recommend you try downloading and installing version 0.22.0: http://hadoop.apache.org/common/releases.html#10+December%2C+2011%3A+release+0.22.0+available

Answer 3

This issue is being tracked at https://issues.apache.org/jira/browse/HADOOP-7682

Answer 4

Use this change owner of hadoop-Admininstrator folder. You can use chown command for that.

Answer 5

There appears to be a permissions issue related to the path
/tmp/hadoop-Administrator/mapred/local/taskTracker
as evidenced by the error message

ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.io.IOException: Failed to set permissions of path: /tmp/hadoop-Administrator/mapred/local/taskTracker

The account the taskTracker is being started under needs the ability to chmod the specified folder. It may need more control, such as being owner, for other aspects. I don't recall the specific permissions required for components in hadoop setup.

I haven't dealt with the permission setup aspect of Hadoop much, especially on windows (at all), so what I'm saying is based heavily on the error message you've provided. I also haven't dealth with cygwin folder permission, so I don't know the solution to correct it, but hopefully this will get you pointed in the right direction.