How to build a reactive pipeline with different Publishers without losing data?

Question

Imagine you have code like this:

public List<Group> addUserToGroups(String username, String label) {
  Mono<User> userMono = webClient.getUser(username);
  User user = userMono.block();

  Flux<Group> groupsFlux = webClient.getGroups(label);
  List<Group> groups = groupsFlux.collectList().block();

  groups.forEach(group -> 
      webClient.addUserToGroup(user.getId(), group.getId()).block()
  );

  return groups;
}

But now you want to refactor this code into a non-blocking reactive pipeline, and the main method to return a Flux<Group> .

So maybe you would start doing something like this:

public Flux<Group> addUserToGroups(String username, String label) {
  return webClient.getUser(username)
      .flatMapMany(user -> webClient.getGroups(label))
      ...
}

But now we have a problem, the values in the resulting Flux are Group and we need the User info, which we lost, in the next step.

Therefore the wished pipeline data-flow could be represented like that:

start
|
U
| /
|/
G1,U
| \
|  UG1----|
|         | 
G2,U      G1
| \       |
|  UG2----|
|         |
          G2
          |
        result: G1, G2

UGn is the result of calling webClient.addUserToGroup

What would be the proper way to implement this?

Answer 1

In general, whenever you feel the need to map to another value, but also keep the value you had already, you've got three basic options:

• Use nested flatMap() calls;
• Create a new object to hold both types and use composition (or use a Tuple );
• Change your object structure or underlying service so the "mapped" object holds all the fields you need.

The advantage to nested flatMap() calls is that it's quick and easy, but the disadvantage is that the code can become near enough unreadable with lots of levels of nesting. Composition solves that, but then you obviously need to create new types (or use Tuple types, which are obviously less descriptive.)

Changing the object structure is the best of both worlds if it makes sense to do so , so it's worth considering - but most of the time, in my experience, it's not feasible or sensible.

Using a nested call here would be fine IMHO, since it's only a single level of nesting - something like:

return webClient.getUser(username)
        .flatMapMany(
                user -> webClient.getGroups(label)
                        .flatMap(group -> webClient.addUserToGroup(user.getId(), group.getId()).thenReturn(group))
        )
        .collectList();

Not necessarily related to the question, but there's a second aspect that I don't particularly like with this method - it's actually doing two things - it's retrieving the groups for a user, and then firing off a second call to add the user to a group (functionality there seems a bit odd, but I don't know the use case.) I'd advise separating those two pieces of functionality into two separate methods if you are able.