Call a WebService and a REST API using JDK8 Streams and CompletableFuture

Question

I have a SOAP call that I need to make and then process the results from the SOAP call in a REST call. Each set of calls is based on a batch of records. I am getting completely lost in trying to get this to run using JDK8 streams as asynchronous as possible. How can I accomplish this?

SOAP Call:

CompletableFuture<Stream<Product>> getProducts(final Set<String> criteria)
{
    return supplyAsync(() -> {
        ...
        return service.findProducts(request);
    }, EXECUTOR_THREAD_POOL);
}

REST Call:

final CompletableFuture<Stream<Result>> validateProducts(final Stream<Product> products)
{
    return supplyAsync(() -> service
        .submitProducts(products, false)
        .stream(), EXECUTOR_THREAD_POOL);
}

I am trying to invoke the SOAP call, pass the result into the REST call, and collect the results using a JDK8 stream. Each SOAP->REST call is a "set" of records (or batch) similar to paging. (this is totally not working right now but just an example).

@Test
public void should_execute_validations()
{
    final Set<String> samples = generateSamples();

    //Prepare paging...
    final int total = samples.size();
    final int pages = getPages(total);
    log.debug("Items: {} / Pages: {}", total, pages);

    final Stopwatch stopwatch = createStarted();
    final Set<Result> results = range(0, pages)
        .mapToObj(index -> {
            final Set<String> subset = subset(index, samples);
            return getProducts(subset)
                .thenApply(this::validateProducts);
        })
        .flatMap(CompletableFuture::join)
        .collect(toSet());
    log.debug("Executed {} calls in {}", pages, stopwatch.stop());
    assertThat(results, notNullValue());
}

Answer 1

I think there are two usage that are incorrect in your example: thenApply and join .

To chain the 1st call (SOAP) and the 2nd call (REST), you need to use thenCompose instead of thenApply . This is because method "validateProducts" returns completable futures, using "thenApply" will create CompletableFuture<CompletableFuture<Stream<Result>>> in your stream mapping. But what you need is probably CompletableFuture<Stream<Result>> . Using thenCompose can resolve this problem, because it is analogous to "Optional.flatMap" or "Stream.flatMap":

.mapToObj(index -> {
    final Set<String> subset = subset(index, samples);
    return getProducts(subset)
        .thenCompose(this::validateProducts);
})

The 2nd incorrect usage is join. Using join blocks the current thread waiting for the result of that CompletableFuture. In your cases, there are N completable futures, where N is the number of pages. Instead of waiting them one by one, the better solution is to wait all the them use CompletableFuture.allOf(...) . This method returns a new CompletableFuture that is completed when all of the given CompletableFutures complete. So I suggest that you modify your stream usage and return a list of futures. Then, wait the completion. And finally, retrieve the results:

List<CompletableFuture<Stream<Result>>> futures = range(0, pages)
    .mapToObj(index -> {
        final Set<String> subset = subset(index, samples);
        return getProducts(subset).thenCompose(this::validateProducts);
    })
    .collect(Collectors.toList());

CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();

for (CompletableFuture<Stream<Result>> cf : futures) {
  // TODO Handle the results and exceptions here
}

You can see the complete program on GitHub .