序列化 avro 生成的 object 到 json 时出现 JsonMappingException

Question

I used avro-tools to generate java classes from avsc files, using:

java.exe -jar avro-tools-1.7.7.jar compile -string schema myfile.avsc 

Then I tried to serialize such objects to json by ObjectMapper, but always got a JsonMappingException saying "not an enum" or "not a union". In my test I create the generated object using it's builder or constructor. I got such exceptions for objects of different classes...

Sample Code:

ObjectMapper serializer = new ObjectMapper(); // com.fasterxml.jackson.databind
serializer.register(new JtsModule()); // com.bedatadriven.jackson.datatype.jts
...
return serializer.writeValueAsBytes(avroConvertedObject); // => JsonMappingException

I also tried many configurations using: serializer.configure(...) but still failed. Versions: Java 1.8, jackson-datatype-jts 2.3, jackson-core 2.6.5, jackson-databind 2.6.5, jackson-annotations 2.6.5

Any suggestions? Thanks!

Answer 1

If the SCHEMA member is really the case (we don't see the full error message), then you can switch it off. I use a mixin to do it, like this:

import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import org.apache.avro.Schema;
import org.junit.Test;

import java.io.File;
import java.io.IOException;

public class AvroGenerTests
{
  abstract class IgnoreSchemaProperty
  {
    // You have to use the correct package for JsonIgnore,
    // fasterxml or codehaus
    @JsonIgnore abstract void getSchema();
  }

  @Test
  public void writeJson() throws IOException {
    BookAvro b = BookAvro.newBuilder()
      .setTitle("Wilk stepowy")
      .setAuthor("Herman Hesse")
      .build();
    ObjectMapper om = new ObjectMapper();
    om.enable(SerializationFeature.INDENT_OUTPUT);
    om.addMixIn(BookAvro.class, IgnoreSchemaProperty.class);
    om.writeValue(new File("plik_z_gen.json"), b);
  }
}

Answer 2

The previous post answers the question correctly. I am just adding on to the previous answer. Instead of writing it to a file I converted it to a string before sending it as body in a POST request.

public class AvroGenerateJSON
{
  abstract class IgnoreSchemaProperty
  {
    // You have to use the correct package for JsonIgnore,
    // fasterxml or codehaus
    @JsonIgnore abstract void getSchema();
  }

  public String convertToJson() throws IOException {
    BookAvro b = BookAvro.newBuilder()
      .setTitle("Wilk stepowy")
      .setAuthor("Herman Hesse")
      .build();
    ObjectMapper om = new ObjectMapper();
    om.enable(SerializationFeature.INDENT_OUTPUT);
    om.addMixIn(BookAvro.class, IgnoreSchemaProperty.class);
    String jsonString = om.writeValueAsString(b);
    return jsonString;
  }
}

Answer 3

After finding the code example at https://www.programcreek.com/java-api-examples/?api=org.apache.avro.io.JsonEncoder I wrote a method that should convert any given Avro object (they extend GenericRecord) to a Json String. Code:

import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.io.DatumWriter;
import org.apache.avro.io.EncoderFactory;
import org.apache.avro.io.JsonEncoder;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;

// ... Class header etc. ...

public static <T extends GenericRecord> String convertToJsonString(T event) throws IOException {

    String jsonstring = "";

    try {
        DatumWriter<T> writer = new GenericDatumWriter<T>(event.getSchema());
        OutputStream out = new ByteArrayOutputStream();
        JsonEncoder encoder = EncoderFactory.get().jsonEncoder(event.getSchema(), out);
        writer.write(event, encoder);
        encoder.flush();
        jsonstring = out.toString();
    } catch (IOException e) {
        log.error("IOException occurred.", e);
        throw e;
    }

    return jsonstring;
}

Answer 4

My req'ts got changed on me and I was told I needed to convert Avro objects straight to JSON without preserving any of the meta-data. My other answer herein that specified a method convertToJsonString converts the entire Avro object to JSON so that using a de-encoder you can re-create the original Avro object as an Avro object. That isn't what my mgt. wanted anymore so I was back to the old drawing board.

As a Hail Mary pass I tried using Gson and it works to do what I now had to do. It's very simple:

 Gson gson = new Gson();
 String theJsonString = gson.toJson(object_ur_converting);

And you're done.

Answer 5

Agree with Shivansh's answer. To add, there might be instances where we need to use the avro-generated pojo in other classes. Under the hood, spring uses jackson library in handling this so we need to override global jackson config by adding a class

@Configuration
public class JacksonConfiguration {
    public abstract IgnoreSchemaProperty {
          @JsonIgnore abstract void getSchema();
    }
    @Bean
    public ObjectMapper objectMapper() {
          ObjectMapper om = new ObjectMapper();
          om.addMixIn(SpecificRecordBase.class, IgnoreSchemaProperty.class);
          return om;
    }
}

SpecificRecordBase -if we want to ignore the schema field from all avro generated classes. In this way, we can serialize/deserialize our avro classes and use it anywhere in our application without getting the issue.

Answer 6

2022 Avro field names

abstract class IgnoreSchemaPropertyConfig {
   // You have to use the correct package for JsonIgnore,
   // fasterxml or codehaus
   @JsonIgnore
   abstract void getClassSchema();

   @JsonIgnore
   abstract void getSpecificData();

   @JsonIgnore
   abstract void get();

   @JsonIgnore
   abstract void getSchema();
}