Json to avro conversion

Question

I'm converting Json to avro. I have json data in JSONArray . So while converting it into byte array i'm facing the problem.

below is my code:

static byte [] fromJsonToAvro(JSONArray json, String schemastr) throws Exception {

ExcelToJson ejj = new ExcelToJson();
List<String> list = new ArrayList<String>();


if (json != null) { 
    int len = json.length();
    for (int i=0;i<len;i++){ 
        list.add(json.get(i).toString());
    } 
}


InputStream input = new ByteArrayInputStream(list.getBytes()); //json.toString().getBytes()

 DataInputStream din = new DataInputStream(input); 
                  .
                  . 
                  .//rest of the logic

So how can i do it? How to convert JsonArray object to bytes(ie, how to use getBytes() method for JsonArray objects). The above code giving an error at list.getBytes() and saying getBytes() is undifined for list.

Answer 1

Avro works at the record level, bound to a schema. I don't think there's such a concept as "convert this JSON fragment to bytes for an Avro field independent of any schema or record".

Assuming the array is part of a larger JSON record, if you're starting with a string of the record, you could do

public static byte[] jsonToAvro(String json, String schemaStr) throws IOException {
    InputStream input = null;
    DataFileWriter<GenericRecord> writer = null;
    Encoder encoder = null;
    ByteArrayOutputStream output = null;
    try {
        Schema schema = new Schema.Parser().parse(schemaStr);
        DatumReader<GenericRecord> reader = new GenericDatumReader<GenericRecord>(schema);
        input = new ByteArrayInputStream(json.getBytes());
        output = new ByteArrayOutputStream();
        DataInputStream din = new DataInputStream(input);
        writer = new DataFileWriter<GenericRecord>(new GenericDatumWriter<GenericRecord>());
        writer.create(schema, output);
        Decoder decoder = DecoderFactory.get().jsonDecoder(schema, din);
        GenericRecord datum;
        while (true) {
            try {
                datum = reader.read(null, decoder);
            } catch (EOFException eofe) {
                break;
            }
            writer.append(datum);
        }
        writer.flush();
        return output.toByteArray();
    } finally {
        try { input.close(); } catch (Exception e) { }
    }
}

Answer 2

For an on-line json to avro converter check the following URL

http://avro4s-ui.landoop.com

It is using the library avro4s that offers a lot of conversions including json=>avro

Answer 3

This discussion is likely useful:

http://mail-archives.apache.org/mod_mbox/avro-user/201209.mbox/%3CCALEq1Z8s1sfaAVB7YE2rpZ=v3q1V_h7Vm39h0HsOzxJ+qfQRSg@mail.gmail.com%3E

The gist is that there is a special Json schema and you can use JsonReader/Writer to get to and from that. The Json schema you should use is defined here:

https://github.com/apache/avro/blob/trunk/share/schemas/org/apache/avro/data/Json.avsc