How to run MR job in Normal privilege

跟風遠走 提交于 2019-12-13 03:35:49

问题


I have installed Hadoop 2.3.0 and able to execute MR jobs successfully. But when I trying to execute MR jobs in normal privilege (without admin privilege) means job get fails with following exception. I tried "WordCount.jar" sample.

14/10/28 09:16:12 INFO mapreduce.Job: Task Id : attempt_1414467725299_0002_r_000
000_1, Status : FAILED
Error: java.lang.NullPointerException
        at org.apache.hadoop.mapred.Task.getFsStatistics(Task.java:347)
        at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(Re
duceTask.java:478)
        at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:414
)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1548)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

By debugging the source I drill down the problem occurs at class YarnChild.java

childUGI.doAs(new PrivilegedExceptionAction<Object>() {
                @Override
                public Object run() throws Exception {
                    // use job-specified working directory
                    FileSystem.get(job).setWorkingDirectory(job.getWorkingDirectory());
                    taskFinal.run(job, umbilical); // run the task
                    return null;
                }
            });

But If I start "NodeManager" with admin privilege mean the above exception won't occur. I don't know why MR job not working when I start "NodeManager" with normal privilege.

If anyone know the reason and solution for above problem. Please do me the favor as soon as possible.

来源:https://stackoverflow.com/questions/26600916/how-to-run-mr-job-in-normal-privilege

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!