问题
I am trying to make a custom partitioner to allocate each unique key to a single reducer. this was after the default HashPartioner failed Alternative to the default hashpartioner provided with hadoop
I keep getting the following error. It has something to do with the constructor not receiving its arguments, from what I can tell from doing some research. but in this context, with hadoop, aren't the arguments passed automatically by the framework? I cant find an error in the code
18/04/20 17:06:51 INFO mapred.JobClient: Task Id : attempt_201804201340_0007_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.NoSuchMethodException: biA3pipepart$parti.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:587)
This is my partitioner:
public class Parti extends Partitioner<Text, Text> {
String partititonkey;
int result=0;
@Override
public int getPartition(Text key, Text value, int numPartitions) {
String partitionKey = key.toString();
if(numPartitions >= 9){
if(partitionKey.charAt(0) =='0' ){
if(partitionKey.charAt(2)=='0' )
result= 0;
else
if(partitionKey.charAt(2)=='1' )
result= 1;
else
result= 2;
}else
if(partitionKey.charAt(0)=='1'){
if(partitionKey.charAt(2)=='0' )
result= 3;
else
if(partitionKey.charAt(2)=='1' )
result= 4;
else
result= 5;
}else
if(partitionKey.charAt(0)=='2' ){
if(partitionKey.charAt(2)=='0' )
result= 6;
else
if(partitionKey.charAt(2)=='1' )
result= 7;
else
result= 8;
}
} //
else
result= 0;
return result;
}// close method
}// close class
My mapper signature
public static class JoinsMap extends Mapper<LongWritable,Text,Text,Text>{
public void Map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{
My reducer signiture
public static class JoinsReduce extends Reducer<Text,Text,Text,Text>{
public void Reduce (Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
main class:
public static void main( String[] args ) throws Exception {
Configuration conf1 = new Configuration();
Job job1 = new Job(conf1, "biA3pipepart");
job1.setJarByClass(biA3pipepart.class);
job1.setNumReduceTasks(9);//***
job1.setOutputKeyClass(Text.class);
job1.setOutputValueClass(Text.class);
job1.setMapperClass(JoinsMap.class);
job1.setReducerClass(JoinsReduce.class);
job1.setInputFormatClass(TextInputFormat.class);
job1.setOutputFormatClass(TextOutputFormat.class);
job1.setPartitionerClass(Parti.class); //+++
// inputs to map.
FileInputFormat.addInputPath(job1, new Path(args[0]));
// single output from reducer.
FileOutputFormat.setOutputPath(job1, new Path(args[1]));
job1.waitForCompletion(true);
}
keys emitted by Mapper are the following:
0,0
0,1
0,2
1,0
1,1
1,2
2,0
2,1
2,2
and the Reducer only writes keys and values it receives.
回答1:
SOLVED
I just added static
to my Parti
class like the mapper and reducer classes as suggested by comment (user238607).
public static class Parti extends Partitioner<Text, Text> {
来源:https://stackoverflow.com/questions/49944554/custom-partitioner-in-hadoop-error-java-lang-nosuchmethodexception-init