How to do unit testing of custom RecordReader and InputFormat classes?

后端 未结 2 603
失恋的感觉
失恋的感觉 2021-01-14 14:38

I have developed one map-reduce program. I have written custom RecordReader and InputFormat classes.

I am using MR Unit and <

2条回答
  •  南笙
    南笙 (楼主)
    2021-01-14 15:05

    thanks to user7610

    compiled and somewhat tested version of the example code from the answer

    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.mapreduce.InputFormat;
    import org.apache.hadoop.mapreduce.RecordReader;
    import org.apache.hadoop.mapreduce.TaskAttemptContext;
    import org.apache.hadoop.mapreduce.TaskAttemptID;
    import org.apache.hadoop.mapreduce.lib.input.FileSplit;
    import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
    import org.apache.hadoop.util.ReflectionUtils;
    import java.io.File;
    
    Configuration conf = new Configuration(false);
    conf.set("fs.default.name", "file:///");
    
    File testFile = new File("path/to/file");
    Path path = new Path(testFile.getAbsoluteFile().toURI());
    FileSplit split = new FileSplit(path, 0, testFile.length(), null);
    
    InputFormat inputFormat = ReflectionUtils.newInstance(MyInputFormat.class, conf);
    TaskAttemptContext context = new TaskAttemptContextImpl(conf, new TaskAttemptID());
    RecordReader reader = inputFormat.createRecordReader(split, context);
    
    reader.initialize(split, context);
    

提交回复
热议问题