将 Avro 二进制字符串转换为 Json

Question

I have a String in Avro Binary Format. I want to convert the String to json. Can someone please guide me?. I tried using the solutions available online It did not work.

public String avroToJson(byte[] avro) throws IOException {
        boolean pretty = false;
        GenericDatumReader<GenericRecord> reader = null;
        JsonEncoder encoder = null;
        ByteArrayOutputStream output = null;
        try {
            reader = new GenericDatumReader<GenericRecord>();
            InputStream input = new ByteArrayInputStream(avro);
            DataFileStream<GenericRecord> streamReader = new DataFileStream<GenericRecord>(input, reader);
            output = new ByteArrayOutputStream();
            Schema schema = streamReader.getSchema();
            DatumWriter<GenericRecord> writer = new GenericDatumWriter<GenericRecord>(schema);
            encoder = EncoderFactory.get().jsonEncoder(schema, output, pretty);
            for (GenericRecord datum : streamReader) {
                writer.write(datum, encoder);
            }
            encoder.flush();
            output.flush();
            return new String(output.toByteArray());
        } finally {
            try {
                if (output != null) output.close();
            } catch (Exception e) {
            }
        }
    }

I am converting the my String to byte array using getBytes() and passing it to this function. I am getting this exception. Exception in thread "main" org.apache.avro.InvalidAvroMagicException: Not an Avro data file.

Answer 1

Avro specifies a binary format for serializing one object, but also a Object Container File (also known as data file) which can hold many objects in a useful way for file access.

DataFileStream expects the container file, but from your description, it looks like you have a single serialized instance.

You probably want something like:

  public String avroToJson(Schema schema, byte[] avroBinary) throws IOException {
    // byte to datum
    DatumReader<Object> datumReader = new GenericDatumReader<>(schema);
    Decoder decoder = DecoderFactory.get().binaryDecoder(avroBinary, null);
    Object avroDatum = datumReader.read(null, decoder);

    // datum to json
    String json = null;
    try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
      DatumWriter<Object> writer = new GenericDatumWriter<>(schema);
      JsonEncoder encoder = EncoderFactory.get().jsonEncoder(schema, baos, false);
      writer.write(avroDatum, encoder);
      encoder.flush();
      baos.flush();
      return new String(baos.toByteArray(), StandardCharsets.UTF_8);
    }
  }

Note that this means you must know the schema in advance to deserialize the binary data. If it were an Avro data file, you could get the schema from the file metadata.

Answer 2

On this line: Object avroDatum = datumReader.read(null, decoder);

I am getting exception:org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -1

Did anyone successfully convert AVRO binary to JSON > Please share.

Answer 3

With generics:

public <T extends SpecificRecordBase> String convertAvroToJson(T payload) throws Exception {

    DatumWriter<T> writer = new SpecificDatumWriter<T>(payload.getSchema());
    ByteArrayOutputStream stream = new ByteArrayOutputStream();
    byte[] payloadJsonBytes;

    try{

        Encoder jsonEncoder = EncoderFactory.get().jsonEncoder(payload.getSchema(), stream);
        writer.write(payload, jsonEncoder);
        jsonEncoder.flush();
        payloadJsonBytes = stream.toByteArray();

    } catch (IOException e) {
        throw new Exception("Cannot convert avro to JSON! ", e);
    }

    String payloadJson = new String(payloadJsonBytes, StandardCharsets.UTF_8);

    return payloadJson;
}