Using AutoValueSchema in Apache Beam PCollection gives `RuntimeException: Creator parameter arg0 Doesn't correspond to a schema field`

问题

I am trying to have a PCollection of AutoValue-defined objects that I have created, and I've added the appropriate annotations to infer the Schema via DefaultSchema(AutoValueSchema.class). Like so:

@DefaultSchema(AutoValueSchema.class)
@AutoValue
public abstract class MyAutoClas {
  public abstract String getMyStr();
  public abstract Integer getMyInt();

  @CreateSchema
  public static MyAutoClass create(String myStr, Integer myInt) {
    return new AutoValue_MyAutoClass(myStr, myInt);
  }
}

I have a small test case that looks like this:

PCollection<KV<String, MyAutoClass>> result = pipeline
    .apply(Create.of(MyAutoClass.create("abc", 1)))
    .apply(WithKeys.of(in -> in.getMyStr()));

PAssert.that(result).containsInAnyOrder(KV.of("abc", MyAutoClass.create("abc", 1)));
pipeline.run().waitUntilFinish();

When I try to run this, I am seeing the following errors:

[ERROR] testMyAutoValueClass(.....)  Time elapsed: 1.891 s  <<< ERROR!
java.lang.RuntimeException: Creator parameter arg0 Doesn't correspond to a schema field
    at org.apache.beam.sdk.schemas.utils.ByteBuddyUtils$InvokeUserCreateInstruction.<init>(ByteBuddyUtils.java:717)
    at org.apache.beam.sdk.schemas.utils.ByteBuddyUtils$StaticFactoryMethodInstruction.<init>(ByteBuddyUtils.java:660)
    at org.apache.beam.sdk.schemas.utils.JavaBeanUtils.createStaticCreator(JavaBeanUtils.java:284)
    at org.apache.beam.sdk.schemas.utils.JavaBeanUtils.lambda$getStaticCreator$4(JavaBeanUtils.java:273)
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1664)
    at org.apache.beam.sdk.schemas.utils.JavaBeanUtils.getStaticCreator(JavaBeanUtils.java:269)
    at org.apache.beam.sdk.schemas.AutoValueSchema.lambda$schemaTypeCreatorFactory$673bce5b$1(AutoValueSchema.java:80)
    at org.apache.beam.sdk.schemas.UserTypeCreatorFactory.create(UserTypeCreatorFactory.java:21)
  ....... [ETCETERA] ......

回答1:

This error occurs because ByteBuddy and the Java reflection utilities are not able to infer the parameter names of your @SchemaCreate method (thus complaining about some unknown parameter arg0 - this is the default name).

To ensure Java reflection can find the parameter names, you need to instruct the compiler to include this information. If you are building with Maven, you can do it this way:

<properties>
    <!-- PLUGIN VERSIONS -->
    <maven-compiler-plugin.version>3.1</maven-compiler-plugin.version>

    <!-- OTHER PROPERTIES -->
    <java.version>1.8</java.version>
</properties>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>${maven-compiler-plugin.version}</version>
            <configuration>
                <!-- Original answer -->
                <compilerArgument>-parameters</compilerArgument>
                <!-- Or, if you use the plugin version >= 3.6.2 -->
                <parameters>true</parameters>
                <testCompilerArgument>-parameters</testCompilerArgument>
                <source>${java.version}</source>
                <target>${java.version}</target>
            </configuration>
        </plugin>
    </plugins>
</build>

Note: To accomplish the same thing with Gradle, use this answer.

来源：https://stackoverflow.com/questions/60140366/using-autovalueschema-in-apache-beam-pcollection-gives-runtimeexception-creato

标签

java

apache-beam

auto-value