NoSuchMethodError writing Avro object to HDFS using Builder

问题

I'm getting this exception when writing an object to HDFS:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.avro.Schema$Parser.parse(Ljava/lang/String;[Ljava/lang/String;)Lorg/apache/avro/Schema;
        at com.blah.SomeType.<clinit>(SomeType.java:10)

The line it is referencing in the generated code is this:

public class SomeType extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
  public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse ..........

And the call in my code is this:

val someTypeBuilder = SomeType.newBuilder()
      .setSomeField("some value")
      // set a whole load of other fields

Calling newBuilder() in test code causes no issues at all.

The jar is being run on an HDFS node using the hadoop jar command.

Any ideas what might be going wrong here?

回答1:

org.apache.avroSchema.Parser.parse(String s, String... more) was not introduced until Avro 1.7.5. Notice its absence in the 1.7.4 Schema.Parser documentation. Either update your HDFS node to use a more recent version of Avro, or join the Strings yourself like how the current 1.7 version of Schema does and use the method that takes a single String instead:

public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse(String.join("", prevArg1, prevArg2));

EDIT: I've never tried upgrading Avro on a Hadoop installation, but I would guess you could probably just find the jars and replace them manually if it's a minor version upgrade.

~/dev/hadoop/hadoop-2.7.1$ find . -name '*avro*.jar'
./share/hadoop/tools/lib/avro-1.7.4.jar
./share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/avro-1.7.4.jar
./share/hadoop/common/lib/avro-1.7.4.jar
./share/hadoop/mapreduce/lib/avro-1.7.4.jar
./share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/avro-1.7.4.jar

Try downloading Avro 1.7.7 and replacing all those found jars with downloaded one. I would also back up the Hadoop installation in case it goes awry.

来源：https://stackoverflow.com/questions/31687550/nosuchmethoderror-writing-avro-object-to-hdfs-using-builder

标签

java

scala

Hadoop

HDFS

avro