Hadoop gen1 vs Hadoop gen2

Question

I am a bit confused about place of tasktracker in Hadoop-2.x.

Daemons in Hadoop-1.x are namenode, datanode, jobtracker, taskracker and secondaryna

Answer 1

namenode, datanode, resourcemanager, applicationmaster

You missed another daemons in Hadoop-2.x from above list which is NodeManager. This daemon runs on the individual nodes like tasktracker. On startup, this component registers with the RM and sends information about the resources available on the nodes. Subsequent NM-RM communication is to provide updates on container statuses – new containers running on the node, completed containers, etc.

So here is what happen. RM allocates resources to job. one of the allocated node act like applicationmaster and communicate with other nodes. In simple terms now you can consider application master is jobtracker and all others are tasktraker nodes. RM is free to service other users for more jobs. Now that is the beauty of the MR v2 that you can run multiple MR jobs as well as other applications like Spark jobs on the same cluster. ResourceManage is responsible for management of the cluster and spin allocate resources or nodes for jobs and one of the allocated node becomes application master.

Shahzad