问题
Hadoop execute map, shuffle/sort and reduce phase. I want to know the duration of each phase. I mean, how long do they take?
I have searched lots of documents but I couldn't find any clear solution. For example, I set the configure file and enable the profiler in order to use Hadoop profiler. But the result file "java.hprof.txt" is not generated. Furthermore, I'm not sure the result file includes the information what I want.
Can you help me?
回答1:
One way of tracking your job's progress is looking at the Web UI provided by Hadoop. By default it is available at http://server-adress:50030/jobtracker.jsp.
There you can find information about task runtimes, task progress, view logs and many other userful tools.
回答2:
You can find all this information in the jobtracker's job history log, available in its web UI. It is different than the UI you see when the job is running. There, you will find the times for each phase separately.
来源：https://stackoverflow.com/questions/22188887/how-can-i-measure-the-duration-of-each-phasemap-shuffle-sort-reduce-in-hadoo