如何在Avro架构中定义复杂类型

Question

I have reviewed avro documentation as well as several examples online (and similar StackOverflow questions). I then attempted to define an avro schema, and had to progressively back out fields to determine what my issue was (the error message from the avro library in python was not as helpful as one would hope). I have a JSON document that I would like to convert to Avro and I need a schema to be specified for that purpose (using avro-tools to generate the schema from the json did not work as expected and yielded an AvroTypeException when attempting to convert the json into avro). I am using Avro version 1.7.7. Here is the JSON document for which I would like to define the avro schema:

{
  "method": "Do_Thing",
  "code": 200,
  "reason": "OK",
  "siteId": {
    "string": "a1283632-121a-4a3f-9560-7b73830f94j8"
  }
}

I was able to define the schema for the non-complex types but not for the complex "siteId" field:

{
  "namespace" : "com.example",
  "name" : "methodEvent",
  "type" :  "record",
  "fields" : [
    {"name": "method", "type": "string"},
    {"name": "code", "type": "int"},
    {"name": "reason", "type": "string"}
    {"name": "siteId", "type": [ "null", "string" ]}
  ]
}

Attempting to use the previous schema to convert the Json object to avro yields an avro.io.AvroTypeException: The datum [See JSON Object above] is not an example of the schema [See Avro Schema Object above]. I only see this error when attempting to define a field in the schema to represent the "siteId" field in the above json.

Answer 1

Avro's python implementation represents unions differently than their JSON encoding: it "unwraps" them, so the siteId field is expected to be just the string, without the wrapping object. See below for a few examples.

Valid JSON encodings

Non-null siteid :

{
  "method": "Do_Thing",
  "code": 200,
  "reason": "OK",
  "siteId": {
    "string": "a1283632-121a-4a3f-9560-7b73830f94j8"
  }
}

Null siteid :

{
  "method": "Do_Thing",
  "code": 200,
  "reason": "OK",
  "siteId": null
}

Valid python objects (in-memory representation)

Non-null siteid :

{
  "method": "Do_Thing",
  "code": 200,
  "reason": "OK",
  "siteId": "a1283632-121a-4a3f-9560-7b73830f94j8"
}

Null siteid :

{
  "method": "Do_Thing",
  "code": 200,
  "reason": "OK",
  "siteId": null
}

Note that null s are unwrapped in both cases which is why your solution isn't working.

Unfortunately, the python implementation doesn't have a JSON decoder/encoder currently (AFAIK), so there is no easy way to translate between the two representations. Depending on the source of your JSON-encoded data, the simplest might be to edit it to not wrap union instances anymore.

Answer 2

I was able to resolve the issue with the following schema:

{
  "namespace" : "com.example",
  "name" : "methodEvent",
  "type" :  "record",
  "fields" : [
    {"name": "method", "type": "string"},
    {"name": "code", "type": "int"},
    {"name": "reason", "type": "string"}
    {
      "name": "siteId", 
      "type": {
        "name" : "siteId",
        "type" : "record",
        "fields" : [
          "name" : "string",
          "type" : [ "null", "string" ]
        ]
      }
    },
    "default" : null
  ]
}