Jackson deserialize Flux of arrays

Question

I'm trying to deserialize a Flux of List<CustomObject>. In this case, the grouping of CustomObjects is necessary: each List is a sequence and my later processing needs to take into account the separate sequences.

I've looked at this question which dealt with an array of objects. In fact, I've been able to use their Mono solution approach to retrieve a Mono<List<List>>. This works, but seems quite inefficient. I'd like to start processing each List, rather than requiring the Mono to complete retrieval of the full bulk set. Our full set could be in the realm of 20K or more lists, some of which we're going to discard, so earlier processing would be desirable.

What happens when I try using Flux instead of Mono: I've tried this using both WebClient and Feign - same effects in each. Stepping through the Jackson code, it never pulls back TokenBuffers when dealing with the Flux world. That seems in line with what's in the Spring documentation for Codecs and Jackson JSON - "when decoding to a multi-value publisher (eg Flux), each TokenBuffer is passed to the ObjectMapper as soon as enough bytes are received for a fully formed object"

Thoughts? Even better, working examples?

Answer 1

In the case of streaming mimetype such as application/x-ndjson , WebFlux will serialize to JSON and flush in the network each item of the Flux .

@GetMapping(path = "/customObjects", produces = MediaType.APPLICATION_NDJSON_VALUE)
public Flux<Response> streamCustomObjects() {
return Flux.interval(Duration.ofSeconds(1))
                .map(e -> new Response(List.of(new CustomObject("George", 20), new CustomObject("Lefteris", 28))));
}

The following WebClient call produces a Flux of Responses :

Flux<Response> responses = webClient.get()
      .uri("/customObjects")
      .retrieve()
      .bodyToFlux(Response.class);

responses.subscribe(System.out::println);

Sample output:

(after 1 sec)
    Response(customObjects=[CustomObject(name=George, age=20), CustomObject(name=Lefteris, age=28)])
(after 2 sec)
    Response(customObjects=[CustomObject(name=George, age=20), CustomObject(name=Lefteris, age=28)])
(after 3 sec)
    Response(customObjects=[CustomObject(name=George, age=20), CustomObject(name=Lefteris, age=28)])

Answer 2

Did try the posted proposed solution of x-ndjson, but ended up with another issue downstream: Flux + Jackson would then not keep the bounds of the list. Eg, I really needed to know [e1, e2, e3] belong together. I was instead getting onNext lists with single elements in them.

My ultimate resolution ended up being simple: we created an object type with a single field which then contained the list of custom objects. Eg, { sequence: [ e1, e2, e3] }. That then let me receive my ListWrapperObject via Flux, while maintaining the bounds of the sequence for each. Since we didn't own the API, we needed to negotiate with the team we were calling, but thankfully they were able to adjust. I offer that solution to the next reader seeking such a path. I'll also grant that there may have yet been a way to solve it with Jackson should you not be able to amend the API.