Programmatically get progress rate of Hadoop's tasks

问题

For research reasons, I would like to record the progress rate of all tasks in order to analyse the progress rates evolve over time.

I already managed to recompile these JARs in order to log latency of heartbeat packets:

hadoop-yarn-server-common-3.2.0.jar
hadoop-yarn-server-nodemanager-3.2.0.jar
hadoop-yarn-server-resourcemanager-3.2.0.jar

Initially, I thought the progress rate information of each task would be part of the heartbeat packet sent to the ResourceManager. However, by looking at the template of this protocol (RPC), I'm not so sure this information is here.

message NodeHeartbeatRequestProto {
  optional NodeStatusProto node_status = 1;
  optional MasterKeyProto last_known_container_token_master_key = 2;
  optional MasterKeyProto last_known_nm_token_master_key = 3;
  optional NodeLabelsProto nodeLabels = 4;
  repeated LogAggregationReportProto log_aggregation_reports_for_apps = 5;
  repeated AppCollectorDataProto registering_collectors = 6;
  optional NodeAttributesProto nodeAttributes = 7;
  optional uint64 sendingTime = 8;  # related to my latency calculation (ignore it)
}

FYI, this protocol is used in the class NodeHeartbeatRequest.

If I look deep into the source code, it seems the value I am interested in is located in TaskStatus by calling the getProgress() method. But how can I get this information from the heartbeat packet? More precisely, I am working on the nodeHeartbeat method.

It is maybe not in this protocol. The only thing that I am sure is that their is a packet going from workers to master that informs on the progress rate of each task. Does anyone knows where I could find this information?

Apache Hadoop is such a large project and we can easily get lost in its source code. Any help will be gladly appreciated!

来源：https://stackoverflow.com/questions/60849683/programmatically-get-progress-rate-of-hadoops-tasks

标签

Hadoop

yarn