Java “in-order” semaphore

Question

I have an issue and a vague idea for how to fix it, but I'll try to overshare on the context to avoid an XY problem .

I have an asychronous method that immediately returns a guava ListenableFuture that I need to call hundreds of thousands or millions of times (the future itself can take a little while to complete). I can't really change the internals of that method. There's some serious resource contention involved in it internally, so I'd like to limit the number of calls to that method that happen at once. So I tried using a Semaphore :

public class ConcurrentCallsLimiter<In, Out> {
  private final Function<In, ListenableFuture<Out>> fn;
  private final Semaphore semaphore;
  private final Executor releasingExecutor;

  public ConcurrentCallsLimiter(
      int limit, Executor releasingExecutor,
      Function<In, ListenableFuture<Out>> fn) {
    this.semaphore = new Semaphore(limit);
    this.fn = fn;
    this.releasingExecutor = releasingExecutor;
  }

  public ListenableFuture<Out> apply(In in) throws InterruptedException {
    semaphore.acquire();
    ListenableFuture<Out> result = fn.apply(in);
    result.addListener(() -> semaphore.release(), releasingExecutor);
    return result;
  }
}

So then I can just wrap my call in this class and call that instead:

ConcurrentLimiter<Foo, Bar> cl =
    new ConcurrentLimiter(10, executor, someService::turnFooIntoBar);
for (Foo foo : foos) {
  ListenableFuture<Bar> bar = cl.apply(foo);
  // handle bar (no pun intended)
}

This sort of works. The issue is that the tail latency is really bad. Some calls get "unlucky" and end up taking a long time trying to acquire resources within that method call. This is exacerbated by some internal exponential-backoff logic such that the unlucky call gets less and less of a chance to acquire the needed resources, compared to new calls that are more eager and wait a much shorter time before trying again.

What would be ideal to fix it is if there was something similar to that semaphore but that had a notion of order. For instance if the limit is 10, then the 11th call currently must wait for any of the first 10 to complete. What I'd like is for the 11th call to have to wait for the very first call to complete. That way "unlucky" calls aren't continuing to be starved by new calls constantly coming in.

It seems like I could assign an integer sequence number to the calls and somehow keep track of the lowest one yet to finish, but couldn't quite see how to make that work, especially since there's not really any useful "waiting" methods on AtomicInteger or whatever.

Answer 1

You can create semaphore with fairness paramter:

Semaphore(int permits, boolean fair)

//fair - true if this semaphore will guarantee first-in first-out granting of permits under contention, else false

Answer 2

One less-than-ideal solution I thought of was using a Queue to store all of the futures:

public class ConcurrentCallsLimiter<In, Out> {
  private final Function<In, ListenableFuture<Out>> fn;
  private final int limit;
  private final Queue<ListenableFuture<Out>> queue = new ArrayDeque<>();

  public ConcurrentCallsLimiter(int limit, Function<In, ListenableFuture<Out>> fn) {
    this.limit = limit;
    this.fn = fn;
  }

  public ListenableFuture<Out> apply(In in) throws InterruptedException {
    if (queue.size() == limit) {
      queue.remove().get();
    }
    ListenableFuture<Out> result = fn.apply(in);
    queue.add(result);
    return result;
  }
}

However, this seems like a big waste of memory. The objects involved could be a bit big and the limit could be set pretty high. So I'm open to better answers that don't have O(n) memory usage. It seems like it should be doable in O(1).

Answer 3

"if there was something similar to that semaphore but that had a notion of order." There exists such a beast. It is called Blocking queue . Create such a queue, put 10 items there, and use take instead of acquire and put instead of release .