帶有嵌套映射的動態模式在 avro 中帶有數組 - 獲取 ClassCastException

Question

我正在嘗試使用數組序列化嵌套映射。

[
  {
    "valueMap":{
    "num": 3646,
    "pos": {
      "type": "Point",
      "coord": [
        32.234756,
        34.12345
      ]
    }
  },
    "id": "12345",
    "ph": "4546747474"
  }
]

下面是生成的架構：

{
  "type": "record",
  "name": "test",
  "namespace": "test",
  "doc": "info",
  "fields": [
    {
      "name": "id",
      "type": "string"
    },
    {
      "name": "ph",
      "type": "string"
    },
    {
      "name": "valueMap",
      "type": {
        "type": "array",
        "items": {
          "type": "record",
          "name": "valueMap",
          "namespace": "valueMap",
          "doc": "valueMap",
          "fields": [
            {
              "name": "num",
              "type": {
                "type": "long",
                "logicalType": "num"
              },
              "doc": "num"
            },
            {
              "name": "pos",
              "type": {
                "type": "map",
                "values": {
                  "type": "map",
                  "values": {
                    "type": "record",
                    "name": "pos",
                    "namespace": "pos",
                    "doc": "pos",
                    "fields": [
                      {
                        "name": "type",
                        "type": [
                          "null",
                          "string"
                        ],
                        "doc": "type"
                      },
                      {
                        "name": "coord",
                        "type": [
                          "null",
                          {
                            "type": "array",
                            "items": "double"
                          }
                        ],
                        "doc": "coord"
                      }
                    ]
                  }
                }
              },
              "doc": "pos"
            }
          ]
        }
      },
      "doc": "valueMap_docs_fieldNames"
    }
  ]
}

這是它不喜歡上面的架構和錯誤的代碼片段

java.lang.ClassCastException: 類 java.lang.String 不能轉換為類 java.util.Map

DatumWriter<GenericRecord> genericRecordDatumWriter = new GenericDatumWriter<>(schema);

        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        GenericRecord genericRecord = genericRecordT(schema, message);

        try (DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<>(genericRecordDatumWriter)) {
            dataFileWriter.create(schema, byteArrayOutputStream);
            dataFileWriter.append(genericRecord);
            dataFileWriter.flush();
        } catch (IOException e) {
            log.error("Could not serialize Generic Record.", e);
            throw new AvroRuntimeException("Could not read schema file.");
        }

        return byteArrayOutputStream.toByteArray();
    }

感謝任何幫助。 找不到任何好的例子。

謝謝

我正在使用的代碼

                Schema coordinatesArraySchema = SchemaBuilder.array().items().doubleType();
                //schema for child map 
                Schema xyPosMapSchema = SchemaBuilder.map().values(coordinatesArraySchema);
                xyPosMapSchema.addProp(LOGICAL_TYPE_KEY, "Point");

                //pos fields
                Schema.Field typeF = new Schema.Field("type", Schema.createUnion(Schema.create(Schema.Type.NULL), Schema.create(Schema.Type.STRING)), "type", null);
                Schema.Field coordinateF = new Schema.Field("coordinates", Schema.createUnion(Schema.create(Schema.Type.NULL), coordinatesArraySchema), "coordinates", null);
                List<Schema.Field> xyPosFields = new ArrayList<>();
                xyPosFields.add(typeF);
                xyPosFields.add(coordinateF);
                xyPosMapSchema = Schema.createRecord("xyPos", "xyPos" + "_docs", "xyPos" + "_namespace", false);
                xyPosMapSchema.setFields(xyPosFields);

                //schema for parent map  i.e. measurements
                Schema measurementMapSchema = SchemaBuilder.map().values(xyPosMapSchema);

                measurementFields.add(new Schema.Field(fieldName, Schema.createMap(measurementMapSchema), fieldName, null));
                 Schema measurementSchema = Schema.createRecord(MEASUREMENTS, MEASUREMENTS + "_docs", MEASUREMENTS + "_namespace", false);
                 measurementSchema.setFields(measurementFields);
                return Schema.createArray(measurementSchema);

Answer 1

如果您使用的是 Java，我建議您使用 Avro Maven 插件來生成 Java 類。 這樣，您就無需弄清楚如何構建 GenericRecord。

假設您無論如何都不需要地圖，您可以使用對象的記錄

{
  "type" : "record",
  "name" : "Item",
  "fields" : [ {
    "name" : "valueMap",
    "type" : {
      "type" : "record",
      "name" : "ValueMap",
      "fields" : [ {
        "name" : "num",
        "type" : "int"
      }, {
        "name" : "pos",
        "type" : {
          "type" : "record",
          "name" : "Point",
          "fields" : [ {
            "name" : "type",
            "type" : "string"
          }, {
            "name" : "coord",
            "type" : {
              "type" : "array",
              "items" : "double"
            }
          } ]
        }
      } ]
    }
  }, {
    "name" : "id",
    "type" : "string"
  }, {
    "name" : "ph",
    "type" : "string"
  } ]
}

我從這個 IDL 得到的

protocol MyProtocol {
  record Point {
    string type ;
    array<double> coord ;
  }
  record ValueMap {
    int num ;
    Point pos ;
  }

  record Item {
     ValueMap valueMap;
    string id;
    string ph;
  }
}