How to write JUnit5 or Mockito test for a retry mechanism built using Supplier<CompletableFuture> as a method param?

Question

. 。 Answers to this question are eligible for a +250 reputation bounty. PacificNW_Lover is looking for an . 。

Have a common framework that isn't server specific which we use to import and re-use objects inside our microservices but can be used for any particular Java program which supports Java 8.

Was tasked to create a retry() mechanism that would take the following two parameters:

1. Supplier<CompletableFuture<R>> supplier
2. int numRetries

What I needed it to do is to make a generic component that conducts a retry operation for a CompletableFuture every time there's an exception based on the numRetries specified.

Using Java 8, I wrote the following generic helper class which uses a Supplier to retry a method based on a number of retries specified.

Am new to CompletableFuture , in general, so am wondering how to write a JUnit 5 (and if Mockito is better) test for this method? Am testing for all edge cases and have made this as generic as possible for others to re-use.

Note this is inside an in-house common framework which is imported by other microservices as a Maven dependency but the aim is to be able to re-use this at the Java SE level.

public class RetryUtility {

    private static final Logger log = 
                    LoggerFactory.getLogger(RetryUtility.class);

    public static <R> CompletableFuture<R> retry(
                                       Supplier<CompletableFuture<R>> supplier, 
                                       int numRetries) {
        CompletableFuture<R> completableFuture = supplier.get();
        if (numRetries > 0) {
            for (int i = 0; i < numRetries; i++) {
                completableFuture = 
                            completableFuture
                              .thenApply(CompletableFuture::completedFuture)
                              .exceptionally(
                                  t -> {
                                   log.info("Retrying for {}", t.getMessage());
                                   return supplier.get();
                                })
                              .thenCompose(Function.identity());
            }
        }
        return completableFuture;
    }
}

---

Usage: Presume this code compiles and works, I was able to put in a wrong URL in the config file for client and log.info(Error getting orderResponse = {}); was printed twice via grep against app.log file.

This is the calling class that imports the above class as Maven dependency:

public class OrderServiceFuture {

    private CompletableFuture<OrderReponse> getOrderResponse(
                           OrderInput orderInput,BearerToken token) {
        int numRetries = 1;
        CompletableFuture<OrderReponse> orderResponse =
               RetryUtility.retry(() -> makeOrder(orderInput, token), numRetries);
        orderResponse.join();
        return orderResponse;
    }

    public CompletableFuture<OrderReponse> makeOrder() {
        return client.post(orderInput, token),
            orderReponse -> {
            log.info("Got orderReponse = {}", orderReponse);
        },
        throwable -> {
            log.error("Error getting orderResponse = {}", throwable.getMessage());
        }
    }
}

---

Although this example's calling class uses OrderSerice with an HttpClient making a call, this is a generic utility class that was specifically written to be reusable for any type of method call which returns CompletableFuture<OrderReponse> .

Question(s):

1. How to write a JUnit 5 test case (or Mockito) for this: public static <R> CompletableFuture<R> RetryUtility.retry(Supplier<CompletableFuture<R>> supplier, int numRetries) method?

2. Is there any edge cases or nuances that someone can see with its design and/or implementation?

Answer 1

Before you ask "How do I test method X?"it helps to clarify "What does X actually do?". To answer this latter question I personally like to rewrite methods in a way such that individual steps become clearer.

Doing that for your retry method, I get:

public static <R> CompletableFuture<R> retry(
        Supplier<CompletableFuture<R>> supplier,
        int numRetries
) {
    CompletableFuture<R> completableFuture = supplier.get();
    for (int i = 0; i < numRetries; i++) {
        Function<R, CompletableFuture<R>> function1 = CompletableFuture::completedFuture;
        CompletableFuture<CompletableFuture<R>> tmp1 = completableFuture.thenApply(function1);
        
        Function<Throwable, CompletableFuture<R>> function2 = t -> supplier.get();
        CompletableFuture<CompletableFuture<R>> tmp2 = tmp1.exceptionally(function2);
        
        Function<CompletableFuture<R>, CompletableFuture<R>> function3 = Function.identity();
        completableFuture = tmp2.thenCompose(function3);
    }
    return completableFuture;
}

Note that I removed the unnecessary if statement and also the log.info call. The latter isn't testable unless you pass in a logger instance as a method argument (or make retry non-static and have the logger as an instance variable that is passed via the constructor, or maybe use some dirty hacks like redirecting the logger output stream).

Now, what does retry actually do?

1. It calls supplier.get() once and assigns the value to the completableFuture variable.
2. If numRetries <= 0 , then it returns completableFuture which is the same CF instance that the Supplier's get method returned.
3. If numRetries > 0 , then it does the following steps numRetries times:
   1. It creates a function function1 that returns a completed CF.
   2. It passes function1 to the thenApply method of completableFuture , creating a new CF tmp1 .
   3. It creates another function function2 which is a function that ignores its input argument but calls supplier.get() .
   4. It passes function2 to the exceptionally method of tmp1 , creating a new CF tmp2 .
   5. It creates a third function function3 which is the identity function.
   6. It passes function3 to the thenCompose of tmp2 , creating a new CF and assigning it to the completableFuture variable.

Having a clear breakdown of what a function does lets you see what you can test and what you cannot test and, therefore, might want to refactor. Steps 1. and 2. are very easy to test:

For step 1. you mock a Supplier and test that it's get method is called:

@Test
void testStep1() { // please use more descriptive names...
    Supplier<CompletableFuture<Object>> mock = Mockito.mock(Supplier.class);
    RetryUtility.retry(mock, 0);
    Mockito.verify(mock).get();
    Mockito.verifyNoMoreInteractions(mock);
}

For step 2. you let the supplier return some predefined instance and check that retry returns the same instance:

@Test
void testStep2() {
    CompletableFuture<Object> instance = CompletableFuture.completedFuture("");
    Supplier<CompletableFuture<Object>> supplier = () -> instance;
    CompletableFuture<Object> result = RetryUtility.retry(supplier, 0);
    Assertions.assertSame(instance, result);
}

The tricky part is, of course, step 3. The first thing you should look out for is which variables are influenced by our input arguments. These are: completableFuture , tmp1 , function2 , and tmp2 . In contrast, function1 and function3 are practically constants and are not influenced by our input arguments.

So, how can we influence the listed arguments then? Well, completableFuture is the return value of the get method of our Supplier . If we let get return a mock, we can influence the return value of thenApply . Similarly, if we let thenApply return a mock, we can influence the return value of expectionally . We can apply the same logic to thenCompose and then test that our methods have been called the correct amount of times in the correct order:

@ParameterizedTest
@ValueSource(ints = {0, 1, 2, 3, 5, 10, 100})
void testStep3(int numRetries) {
    CompletableFuture<Object> completableFuture = mock(CompletableFuture.class);
    CompletableFuture<CompletableFuture<Object>> tmp1 = mock(CompletableFuture.class);
    CompletableFuture<CompletableFuture<Object>> tmp2 = mock(CompletableFuture.class);

    doReturn(tmp1).when(completableFuture).thenApply(any(Function.class));
    doReturn(tmp2).when(tmp1).exceptionally(any(Function.class));
    doReturn(completableFuture).when(tmp2).thenCompose(any(Function.class));

    CompletableFuture<Object> retry = RetryUtility.retry(
            () -> completableFuture, // here 'get' returns our mock
            numRetries
    );
    // While we're at it we also test that we get back our initial CF
    Assertions.assertSame(completableFuture, retry);

    InOrder inOrder = Mockito.inOrder(completableFuture, tmp1, tmp2);
    for (int i = 0; i < numRetries; i++) {
        inOrder.verify(completableFuture, times(1)).thenApply(any(Function.class));
        inOrder.verify(tmp1, times(1)).exceptionally(any(Function.class));
        inOrder.verify(tmp2, times(1)).thenCompose(any(Function.class));
    }
    inOrder.verifyNoMoreInteractions();
}

What about the method arguments of thenApply and thenCompose ? There is no way to test that these methods have been called with function1 and function3 , respectively, (unless you refactor your code and move the functions out of that method call) as these functions are local to our method and not influenced by our arguments. But what about function2 ? While we cannot test that exceptionally is called with function2 as an argument, we can test that function2 calls supplier.get() exactly numRetries times. When does it do that? Well, only if the CF fails:

@ParameterizedTest
@ValueSource(ints = {0, 1, 2, 3, 5, 10, 100})
void testStep3_4(int numRetries) {
    CompletableFuture<Object> future = CompletableFuture.failedFuture(new RuntimeException());
    Supplier<CompletableFuture<Object>> supplier = mock(Supplier.class);
    doReturn(future).when(supplier).get();

    Assertions.assertThrows(
            CompletionException.class,
            () -> RetryUtility.retry(supplier, numRetries).join()
    );
    
    // remember: supplier.get() is also called once at the beginning 
    Mockito.verify(supplier, times(numRetries + 1)).get();
    Mockito.verifyNoMoreInteractions(supplier);
}

Similarly, you can test that retry calls the get method n+1 times by providing a supplier, that returns a completed future after it has been called n times.

What we still need to do (and what we should probably have had done as a first step) is to test whether the return value of our method behaves correctly:

• If the CF fails numRetries tries, the CF should complete exceptionally.
• If the CF fails fewer than numRetries tries, the CF should complete normally.

@ParameterizedTest
@ValueSource(ints = {0, 1, 2, 3, 5, 10})
void testFailingCf(int numRetries) {
    RuntimeException exception = new RuntimeException("");
    CompletableFuture<Object> future = CompletableFuture.failedFuture(exception);

    CompletionException completionException = Assertions.assertThrows(
            CompletionException.class,
            () -> RetryUtility.retry(() -> future, numRetries).join()
    );
    Assertions.assertSame(exception, completionException.getCause());
}

@ParameterizedTest
@ValueSource(ints = {0, 1, 2, 3, 5, 10})
void testSucceedingCf(int numRetries) {
    final AtomicInteger counter = new AtomicInteger();
    final String expected = "expected";

    Supplier<CompletableFuture<Object>> supplier =
            () -> (counter.getAndIncrement() < numRetries / 2)
                    ? CompletableFuture.failedFuture(new RuntimeException())
                    : CompletableFuture.completedFuture(expected);

    Object result = RetryUtility.retry(supplier, numRetries).join();

    Assertions.assertEquals(expected, result);
    Assertions.assertEquals(numRetries / 2 + 1, counter.get());
}

Some other cases you might want to consider testing and catching are what happens if numRetries is negative or very large? Should such method calls throw an exception?

---

Another question we haven't touched so far is: Is this the right way to test your code?

Some people might say yes, others would argue that you shouldn't test the internal structure of your method but only its output given some input (ie basically only the last two tests). That's clearly debatable and, like most things, depends on your requirements. (For example, would you test the internal structure of some sorting algorithm, like array assignments and what not?).

As you can see, by testing the internal structure, the tests become quite complicated and involve a lot of mocking. Testing the internal structure can also make refactoring more troublesome, as your tests might start to fail, even though you haven't changed the logic of your method. Personally, in most situations I wouldn't write such tests. However, if a lot of people depended on the correctness of my code, for example on the order in which steps are executed, I might consider it.

In any case, if you choose to go that route, I hope these examples are useful to you.