How to access Hadoop counters values via API?

前端 未结 2 1736
广开言路
广开言路 2021-01-21 19:12

In Hadoop we can increment counter in map/reduce task, it looks like this:

...
context.getCounter(MyCountersEnum.SomeCounter).increment(1);
...
<
2条回答
  •  时光说笑
    2021-01-21 19:49

    Counters represent global counters, defined either by the Map-Reduce framework or applications.

    Each Counter can be of any Enum type. You can define counter as an enum in Driver class

    static enum UpdateCount{
      CNT
     }
    

    And then increment the counter in map/reduce task

    public class CntReducer extends Reducer{
     public void reduce(IntWritable key,Iterable values,Context context)  {
          //do something
          context.getCounter(UpdateCount.CNT).increment(1);
     }
    }
    

    and access them in Driver class

    public int run(String[] args) throws Exception {
     .
     .
     .
     job.setInputFormatClass(TextInputFormat.class);
     job.setOutputFormatClass(TextOutputFormat.class);
     FileInputFormat.setInputPaths(job,in );
     FileOutputFormat.setOutputPath(job, out);
     job.waitForCompletion(true);
     c = job.getCounters().findCounter(UpdateCount.CNT).getValue();
     //Print "c"
     }
    }
    

    c gives the counter value.

    You can find an example here

提交回复
热议问题