问题
I have installed Hadoop 2.3.0 and able to execute MR jobs successfully. But when I trying to execute MR jobs in normal privilege (without admin privilege) means job get fails with following exception. I tried "WordCount.jar" sample.
14/10/28 09:16:12 INFO mapreduce.Job: Task Id : attempt_1414467725299_0002_r_000
000_1, Status : FAILED
Error: java.lang.NullPointerException
at org.apache.hadoop.mapred.Task.getFsStatistics(Task.java:347)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(Re
duceTask.java:478)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:414
)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
By debugging the source I drill down the problem occurs at class YarnChild.java
childUGI.doAs(new PrivilegedExceptionAction<Object>() {
@Override
public Object run() throws Exception {
// use job-specified working directory
FileSystem.get(job).setWorkingDirectory(job.getWorkingDirectory());
taskFinal.run(job, umbilical); // run the task
return null;
}
});
But If I start "NodeManager" with admin privilege mean the above exception won't occur. I don't know why MR job not working when I start "NodeManager" with normal privilege.
If anyone know the reason and solution for above problem. Please do me the favor as soon as possible.
来源:https://stackoverflow.com/questions/26600916/how-to-run-mr-job-in-normal-privilege