如何在 Avro Union 邏輯類型字段中為默認值指定轉換器？

Question

我有以下 Avro 架構：

{
   "namespace":"com.example",
   "type":"record",
   "name":"BuggyRecord",
   "fields":[
      {
         "name":"my_mandatory_date",
         "type":{
            "type":"long",
            "logicalType":"timestamp-millis"
         },
         "default":1502250227187
      },
      {
         "name":"my_optional_date",
         "type":[
            {
               "type":"long",
               "logicalType":"timestamp-millis"
            },
            "null"
         ],
         "default":1502250227187
      }
   ]
}

我使用 maven 生成 Java 文件。 阿羅配置：

       <plugin>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-maven-plugin</artifactId>
            <version>1.8.2</version>
            <executions>
                <execution>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>schema</goal>
                        <!--<goal>idl-protocol</goal>-->
                    </goals>
                    <configuration>
                        <sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
                        <outputDirectory>${project.basedir}/src/main/java</outputDirectory>
                        <enableDecimalLogicalType>true</enableDecimalLogicalType>
                        <stringType>String</stringType>
                    </configuration>
                </execution>
            </executions>
        </plugin>

通過mvn compile代碼生成的類在一些基本代碼上失敗：

@Test
public void Foo(){
    BuggyRecord.Builder buggyRecordBuilder = BuggyRecord.newBuilder();
    buggyRecordBuilder.build();
}

錯誤代碼：

org.apache.avro.AvroRuntimeException: java.lang.ClassCastException: java.lang.Long cannot be cast to org.joda.time.DateTime

    at com.example.BuggyRecord$Builder.build(BuggyRecord.java:301)
    at BuggyRecordTest.Foo(BuggyRecordTest.java:10)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to org.joda.time.DateTime
    at com.example.BuggyRecord$Builder.build(BuggyRecord.java:298)
    ... 23 more

我認為這是由於Converters沒有正確生成：

  private static final org.apache.avro.Conversion<?>[] conversions =
      new org.apache.avro.Conversion<?>[] {
      TIMESTAMP_CONVERSION,
      null,  // <------ THIS ONE IS NOT SET PROPERLY
      null
  };

是我誤用了代碼生成器還是錯誤？

Answer 1

  {
     "name":"my_optional_date",
     "type":[ "null", {"type" : "long", "logicalType": "timestamp-millis"}], "default": null
  }

比解決方案更多的解決方法 - 將“null”類型放在首位，“default”：null

Answer 2

我們遇到了類似的問題，邏輯類型為日期：

{
  "name": "date_field",
  "type": [
    "null",
    {
      "type": "int",
      "logicalType": "date"
    }
  ],
  "default": null
}

我們需要更新到 gradle 插件gradle-avro-plugin的最新版本 ( 0.17.0 ) 和Apache Avro依賴項 ( 1.9.1 )。

同樣在我們的 gradle 構建文件中，我們需要添加

avro {
  dateTimeLogicalType = "JODA"
}

以及對joda-time的依賴

dependencies {
    compile "joda-time:joda-time:2.10.3"
}

gradle-avro-plugin 使用的默認dateTimeLogicalType是 JSR310，但這仍然會導致錯誤（即使我們使用的是 Java 8）。

Answer 3

嘗試在 avro 依賴項和 pom.xml 中的 avro-maven-plugin 中將 avro 的版本升級到 1.9.X

    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.9.1</version>
    </dependency>

    <plugin>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-maven-plugin</artifactId>
        <version>1.9.1</version>
        <executions>
            <execution>
                <phase>generate-sources</phase>
                <goals>
                    <goal>schema</goal>
                    <!--<goal>idl-protocol</goal>-->
                </goals>
                <configuration>
                    <sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
                    <outputDirectory>${project.basedir}/src/main/java</outputDirectory>
                    <enableDecimalLogicalType>true</enableDecimalLogicalType>
                    <stringType>String</stringType>
                </configuration>
            </execution>
        </executions>
    </plugin>

還要確保從 AVRO 模式中刪除以前生成的類並執行 mvn 編譯。