为生产者-消费者模式中的公平数据处理寻找理想的 Azure 服务

Question

I am trying to manage the situation illustrated on the following picture.

We have several clients = producers (also 1 client = 1 to N producers). We need to process the data from each producer fairly (= separately would be the best option), because it can happen that one producer will send us an enormous amount of data and if there were just one queue for all producers, the rest of producers would be blocked. So we need to ensure that no producers will be blocked and served by consumers fairly.

Ideal schema would be that each producer would have own queue and own consumer (asynchronously waiting for any message which will be sent to the queue). This schema is provided by the picture above.

The problem is that number of producers will growth dynamically with number of clients so we need to create queues and consumers dynamically as well.

Moreover, we need to manage another difficulties with data from producers. We need to ensure that older data from Producer A will be processed before the newer one from the same Producer A. (Note: Producers are independent of each other so newer data from Producer B can be processed before older data from Producer A.)

According to my research it would not be any problem to use Azure Queue storage as Queue (it is not a problem to create queues dynamically via queue.CreateIfNotExists(); ). BUT I do not know what to use as proper consumer. I know there is a lot of Azure services, for example: Azure function, Azure WebJob, Azure Event hub… etc.

My question is: What is the best option for this use-case to use as consumers?

We need to serve the queues as fair as possible that no producer's queue will not be blocked by others.

Thanks in advance for any tips!

UPDATE

I was thinking about the use case once again and it results with new schema, see the picture below:

The biggest difference from the previous schema is that there is no 1:1:1 relation between Producer, Queue and Consumer. Each producer does not need own queue and consumer.

There will be just one "master" queue, where producers will send meta messages ("I have sent batch XY into the Table storage A"). There will be also WebJob triggered by the queue and its main task will be sending the information into the "Service bus / event grid / event hub" ( I am just not sure which one would be the best option ).

Service bus / event grid / event hub would trigger the Azure function which would do "consumer" stuff there. It would grab data from Table storage, do some transform and insert it into another structure.

WebJob will also prevent the situation that two micro-batches from the same producer will be processed at the same time. WebJob will postpone the another batches till the last batch will be processed.

Actually instead of service bus / event grid / event hub could be just WebJob which would have some thread pool (consumers) and it would wake up consumer for each producer. Nevertheless I don't think it is the best option for scaling number of customers because resources of the WebJob are not unlimited.

The best option would be one of above-mentioned structures (service bus / event hub / event grid). For example each producer would have own topic in service bus and each topic would trigger own Azure function (which would be consumer).

I am wondering if it is a correct approach?

Answer 1

It's an interesting use case, but in my opinion, Functions will be your best choice for the consumer. I would encapsulate the business logic in a separate file, so all the individual function has is the queue binding and the call to that separate file:

File 1:
public static Task DoStuff(string myQueueItem, ILogger log)
{
    //tranforms here

}

File 2 - n:

[FunctionName("ConsumerA")]
public static void QueueTrigger(
    [QueueTrigger("myqueue-items")] string myQueueItem,
    ILogger log)
{
    await DoStuff(myQueueItem, log);

}

Along with queue creation/deletion, you can use the management API to create and delete functions. Each time you create one you adjust the name and queue name so it binds to the new queue.

You can control parallel processing somewhat by changing the "maxOutstandingRequests" property in the host.json file. On the consumption plan the max requests will be per instance of the app- if you have the setting set to one and the host scales out to three instances then three messages will be processed at a time. Scaling is a little different if you are using a standard App Service plan, but it can still be done dynamically. That dynamic scaling of consumer instances is likely going to be helpful with inconsistent data volumes- the large volume data dumps will be processed more quickly but you won't have to have resources sitting around when things are quiet.

When the host scales out, each instance includes a copy of every function, so increasing throughput due to a large data dump will actually increase the available throughput of your other queues as well instead of decreasing it.