如何将JSON消息转换为具有可空字段的avro架构的有效JSON？

Question

I would like to send a JSON message to a kafka topic with avro-schema.

The avro-schema allows multiple types:

{  
   "name":"typeA",
   "type":[  
      "int",
      "null"
   ],
   "default":null
}

If the value is null, everything is fine. If the type is an int as in this case then this must be specified explicitly. See this ticket AVRO-1582 .

I have this JSON:

{
   "typeA":12345,
   "typeB":[
      {
         "feature":1,
         "value":"1"
      },
      {
         "feature":2,
         "value":"2"
      }
   ],
   "typeC":[
      {
         "a":12345,
         "data":[
            {
               "a":12345,
               "b":[
                  12345,
                  12345,
                  12345
               ]
            }
         ]
      }
   ]
}

I would like to transform into this JSON:

{
   "typeA":{
      "int":12345
   },
   "typeB":{
      "array":[
         {
            "feature":1,
            "value":"1"
         },
         {
            "feature":2,
            "value":"2"
         }
      ]
   },
   "typeC":{
      "array":[
         {
            "a":12345,
            "data":[
               {
                  "a":12345,
                  "b":[
                     12345,
                     12345,
                     12345
                  ]
               }
            ]
         }
      ]
   }
}

Is is possible to transform "typeA":12345 to "typeA":{"int":12345} ? Exists an easy way to handle this issue?

I know the type of every field so I could use a regex in JAVA:

json.replaceAll("typeA\":([^,]*),\"", "typeA\":{\"int\":$1},\"");

It's hard to handle arrays or the last JSON element. How can I solve this problem?

Answer 1

I can transform typeA to:

"typeA":{
    "int":12345
},

But typeB and typeC were too difficult for me, because I couldn't match it precisely. Somehow when I try to replace typeB with a array. Another place get replaced too, which we don't want.

If you or someone else could solve that problem, then typeC can also be easily fixed. Because typeB and typeC are similar. I'm also curious what the solution is. So let me know!

I will now share how I fixed typeA . Here is the Java code:

import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class Test {

    public static void main(String[] args) {
        String line = "{\r\n" +
                "   \"typeA\":12345,\r\n" +
                "   \"typeB\":[\r\n" +
                "      {\r\n" +
                "         \"feature\":1,\r\n" +
                "         \"value\":\"1\"\r\n" +
                "      },\r\n" +
                "      {\r\n" +
                "         \"feature\":2,\r\n" +
                "         \"value\":\"2\"\r\n" +
                "      }\r\n" +
                "   ],\r\n" +
                "   \"typeC\":[\r\n" +
                "      {\r\n" +
                "         \"a\":12345,\r\n" +
                "         \"data\":[\r\n" +
                "            {\r\n" +
                "               \"a\":12345,\r\n" +
                "               \"b\":[\r\n" +
                "                  12345,\r\n" +
                "                  12345,\r\n" +
                "                  12345\r\n" +
                "               ]\r\n" +
                "            }\r\n" +
                "         ]\r\n" +
                "      }\r\n" +
                "   ]\r\n" +
                "}";

        String regex = "(\\\"type[A-Z]\\\"):(\\d*)|(\\[.*)|(.*\\])";
        Pattern pattern = Pattern.compile(regex);
        Matcher matcher = pattern.matcher(line);
        while (matcher.find()) {
             if(matcher.group().equals("\"typeA\":12345")) {
                 String regex2 = "(\\\"typeA\\\"):(\\d*)";
                 line = line.replaceAll(regex2, "$1:{\"int\":$2}");
             }

             if(matcher.group().equals("\"typeB\":") ) {
                 //I couldn't finish this typeB, because it's too difficult
//                 String regex3 = "(\\\"type[B]\\\"):|(\\s{3}],)";
//                 line = line.replaceAll(regex3, "$1 :{ array: $2 ");
             }
        }
         System.out.println("line: " + line);
    }
}

First I used this regex (\\"type[AZ]\\"):(\\d*)|(\\[.*)|(.*\\]) . That regex gives us several groups where we want to look at.

Eventually the while loop encouters "typeA":12345 And that's where we use the regex ("typeA"):(\\d*) . We use that regex to transfrom typeA to:

"typeA":{
    "int":12345
},

Answer 2

This wasn't super fun, due to the names of all the values, but Jackson worked just fine.

I pasted json that you wanted into json2pojo :

{
  "typeA":{
     "int":12345
  },
  "typeB":{
     "array":[
        {
           "feature":1,
           "value":"1"
        },
        {
           "feature":2,
           "value":"2"
        }
     ]
  },
  "typeC":{
     "array":[
        {
           "a":12345,
           "data":[
              {
                 "a":12345,
                 "b":[
                    12345,
                    12345,
                    12345
                 ]
              }
           ]
        }
     ]
  }
}

Used the download as zip feature to get the classes to my local development environment.

Then created this monstrosity of a class that tests out Jackson with the generated classes from json2pojo.

public class JacksonSerialization {

   public static void main(String... args) throws Exception {

       TypeA typeA = new TypeA();
       typeA.setInt(12345);

       TypeB typeB = new TypeB();
       ArrayList<Array> arrays = new ArrayList<>();
       arrays.add(createArray(1, "1"));
       arrays.add(createArray(2, "2"));
       typeB.setArray(arrays);

       TypeC typeC = new TypeC();

       ArrayList<Integer> integers = new ArrayList<>();
       integers.add(12345);
       integers.add(12345);
       integers.add(12345);

       ArrayList<Datum> data = new ArrayList<>();
       Datum datum = new Datum();
       datum.setA(12345);
       datum.setB(integers);
       data.add(datum);

       Array_ array_ = new Array_();
       array_.setA(12345);
       array_.setData(data);

       ArrayList<Array_> array_s = new ArrayList<>();
       array_s.add(array_);
       typeC.setArray(array_s);

       Example example = new Example();
       example.setTypeA(typeA);
       example.setTypeB(typeB);
       example.setTypeC(typeC);

       ObjectMapper mapper = new ObjectMapper();
       mapper.configure(MapperFeature.SORT_PROPERTIES_ALPHABETICALLY, false);
       mapper.configure(SerializationFeature.INDENT_OUTPUT, true);

       ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
       //mapper.writeValue(byteArrayOutputStream, example);
       mapper.writeValue(new File("target/Example.json"), example);

       String json = byteArrayOutputStream.toString();

       json = StringEscapeUtils.escapeCsv(json);

       System.out.println(json);
   }

   private static Array createArray(Integer feature, String value) {

       Array array = new Array();
       array.setFeature(feature);
       array.setValue(value);

       return array;
   }
}

Which when this class runs, produces the following json.

{
  "typeA" : {
    "int" : 123456
  },
  "typeB" : {
    "array" : [ {
      "feature" : 1,
      "value" : "1"
    }, {
      "feature" : 2,
      "value" : "2"
    } ]
  },
  "typeC" : {
    "array" : [ {
      "a" : 12345,
      "data" : [ {
        "a" : 12345,
        "b" : [ 12345, 12345, 12345 ]
      } ]
    } ]
  }
}

Which, I think is fairly close to what you asked for.