控制每秒吞吐量

Question

I have a class witch is responsible for sending data to a client and all others classes use this when they need to send data. Let's call it 'DataSender.class'.

Now the client is asking us to control the throughput to a maximum of 50 calls per second.

I need to create an algoritm on this class (if possible) to keep the number of calls in the current second and if it reaches the maximum of 50, hold the process either with a sleep or something and continue without losing data. Maybe I have to implement a queue or something better than a simple sleep. I need sugestions or a direction to follow.

For the sake of simplicity, just imagine that everyone is using something like this and I cannot change how they call me now. post() return is syncronous but that maybe I can possibly change (not sure yet):

DataSender ds = new DataSender();
ds.setdata(mydata);
if (ds.post()) {
   //data send succesfull
}

Answer 1

If I am not mistaken, what you are looking for is throttling or rate limiting.

As Andrew S pointed out, you will need a Queue, to hold the extra requests and a sender algorithm.

The main point is that because you are not sending the data right away, the callers need to be aware that the data is not necessarily sent when their call returns. Usually senders will not be happy, if their call returns, they assume data is sent, and then the data is lost. There are many reasons why data can be lost in this scenario. As Andrew S pointed out, making senders aware that it will be an asynchronous send queue, maybe with confirmations upon successful send, will be a safer or proper approach.

You will need to decide on the size of the queue, you have to limit the size or you can run out of memory. And you need to decide what happens with the request when queue is full. Also what happens when the end point is not accessible (server down,.network issues, solar flares), keep accepting data to queue or reject / throw exception.

Hint: if you have 50 requests limit, don't blast 50 requests and then sleep for 1 second. Figure out the interval between sends, send one request, make a short interval sleep.

Pro hint: if new data sent invalidates the data that was previously requested to be sent, but not sent yet, you can optimize the data sent by removing the invalidated data from the queue. This is called Conflation. Usual example is stock market prices. Say you got price of ACME 100 ten seconds ago and for whatever reason that data was not sent. If you get a new price for ACME 101 now, it is usually not useful to send the 100 price record, just send the 101 price.