Sending protobuf as JSON in spring-boot

Question

I´m using protobufs with this concrete definition.

message Hash {
    string category = 1;
    repeated KVPair content = 2;
}

message KVPair {
    string key = 1;
    string value = 2;
}

I want to send this as JSON with my spring-boot application. I added this package to my gradle dependencies:

compile group: 'com.google.protobuf', name: 'protobuf-java', version: '3.6.1'

When i try to output Hash generated object with this code:

@RestController
@RequestMapping("/api/crm/")
public class KVController {

    private final KVService kvService;

    public KVController(KVService kvService) {
        this.kvService = kvService;
    }

    @GetMapping("kv/{category}")
    public Hash getHash(@PathVariable String category) {
        Hash hash = kvService.retrieve(category);
        return hash;
    }
}

It throws this ultimate exception:

Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Direct self-reference leading to cycle (through reference chain: com.blaazha.crm.proto.Hash["unknownFields"]->com.google.protobuf.UnknownFieldSet["defaultInstanceForType"]) at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1191) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.BeanPropertyWriter._handleSelfReference(BeanPropertyWriter.java:944) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:721) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.BeanSerial izer.serialize(BeanSerializer.java:155) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913) ~[jackson-data bind-2.9.6.jar:2.9.6] at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:286) ~[spring-web-4.3.18.RELEASE.jar:4.3.18.RELEASE]... 58 common frames omitted

kvService only returns data from redis. It parses Hash data type ( https://redis.io/topics/data-types ) to Hash object defined in proto. Where Hash->category is main key of hash and values in hash redis datatype are converted to KVPair defined in proto. I cannot show all source code, because it calls other systems and source code is very long.

kvService returns valid Hash object, but exception happens when I return this Hash object and spring tries convert it to JSON.

important dependencies in my build.gradle:

def versions = [
        logback: '1.2.3',
        owner: '1.0.10',
        jackson: '2.9.6',

        guava: '25.1-jre',
        guice: '4.2.0',
        grpc: '1.9.1',
        protoc: '3.5.1',

        redis: '2.9.0',
]

dependencies {

compile group: 'ch.qos.logback', name: 'logback-classic', version: versions.logback
compile group: 'org.aeonbits.owner', name: 'owner', version: versions.owner

compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: versions.jackson
compile group: 'com.fasterxml.jackson.core', name: 'jackson-annotations', version: versions.jackson
compile group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: versions.jackson

compile group: 'com.google.guava', name: 'guava', version: versions.guava
compile group: 'com.google.inject', name: 'guice', version: versions.guice
compile group: 'io.grpc', name: 'grpc-netty', version: versions.grpc
compile group: 'io.grpc', name: 'grpc-protobuf', version: versions.grpc
compile group: 'io.grpc', name: 'grpc-stub', version: versions.grpc
compile 'org.glassfish:javax.annotation:10.0-b28'


compile group: 'javax.xml.bind', name: 'jaxb-api', version: '2.2.1'
compile group: 'javax.activation', name: 'activation', version: '1.1.1'

compile group: 'redis.clients', name: 'jedis', version: versions.redis

}

As you can see in my protobuf definition isn´t any self-referencing.

Is there any possible way to fix this problem?

Answer 1

If you're using Spring WebFlux and trying to produces application/json here is what you can do to make it works for all mappings returning protobuf Message type:

@Configuration
@EnableWebFlux
public class WebConfig implements WebFluxConfigurer {

@Override
public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
    configurer.defaultCodecs().jackson2JsonEncoder(
        new Jackson2JsonEncoder(Jackson2ObjectMapperBuilder.json().serializerByType(
                Message.class, new JsonSerializer<Message>() {
                    @Override
                    public void serialize(Message value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
                        String str = JsonFormat.printer().omittingInsignificantWhitespace().print(value);
                        gen.writeRawValue(str);
                    }
                }
        ).build())
    );
}

Answer 2

Class UnknownFieldSet (reached via generated method Hash.getUnknownFields() ) contains getter getDefaultInstanceForType() which returns singleton instance of UnknownFieldSet . This singleton instance references itself in getDefaultInstanceForType() and Jackson-databind can't handle this automatically (see edit2 below).

You might want to use JsonFormat from com.google.protobuf:protobuf-java-util which usescanonical encoding instead of Jackson.

Good luck!

EDIT> For Spring there is ProtobufJsonFormatHttpMessageConverter

EDIT2> Of course you could handle this situation using Mix-in Annotations , but IMHO JsonFormat is definitely the way to go...

Answer 3

To convert the protobuf object to JSON, you should be using the following class from the package com.google.protobuf.util.JsonFormat as:

JsonFormat.printer().print()

Answer 4

I was able to resolve my issue by adding a bean like so in my Spring main application.

Tried a number of answers on StackOverflow with no luck. You may have some success by adding the protobuf types to the registry as defined here, as some of other common answers did not work for me.

@Bean
public Jackson2ObjectMapperBuilderCustomizer jackson2ObjectMapperBuilderCustomizer() {
    JsonFormat.TypeRegistry typeRegistry = JsonFormat.TypeRegistry.newBuilder()
            .add(MyType.getDescriptor())
            .add(MyOtherType.getDescriptor())
            .build();

    JsonFormat.Printer printer = JsonFormat.printer().usingTypeRegistry(typeRegistry)
            .includingDefaultValueFields();
    return o -> o.serializerByType(Message.class, new JsonSerializer<Message>() {
        @Override
        public void serialize(Message message, JsonGenerator gen, SerializerProvider serializers) throws IOException {
            gen.writeRawValue(printer.print(message));
        }
    });
}

Answer 5

An example based on vlp's answer. Do the following:

1. Declare a Bean of type ProtobufJsonFormatHttpMessageConverter . eg

@Configuration
public class AppConfig {
    @Bean
    public ProtobufJsonFormatHttpMessageConverter protobufHttpMessageConverter() {
        return new ProtobufJsonFormatHttpMessageConverter(
                JsonFormat.parser().ignoringUnknownFields(),
                JsonFormat.printer().omittingInsignificantWhitespace()
        );
    }
}

2. Add produces = MediaType.APPLICATION_JSON_VALUE to your controller method. eg

@GetMapping(value = "/get-one-item", produces = MediaType.APPLICATION_JSON_VALUE)
public Item getItems() {
    //Item is a Protobuf message
    return itemService.getOneItem();
}