Azure 服务总线 Azure 函数触发并获取所有消息

Question

I have a queue that a lot of high frequency data is put in. I have an azure function that will trigger when new messages are written to it and write the message to a Azure sql database. My problem is that there are too many messages to write so what I want to do is kind of "accumulate" messages and then write them all at once to the database. I don't see any kind of functionality in Azure function service bus integration to get all(or x) messages and handle them.

Is something like this at all possible?

Any help is appreciated.

Answer 1

My way of solving this so far is to avoid service bus trigger because it involves that the function executes one instance per message. I do a time trigger every 10 seconds and I use the method Receive from the subscription client.

Indeed in this case you would need to manually bring the Azure Service Bus library, create the needed classes and call the Receive method which you can tell how many items you want. Do follow best practice that ASB classes (TopicClient and SubscriptionClient) should be set as singletons because they are expensive objects.

I also have to accumulate messages to sort them and deduplicate them so this is how I solved it. However, it would be very cool to have that kind of functionality out of the box from Azure Functions extension.

Answer 2

从Microsoft.Azure.WebJobs.Extensions.ServiceBus 4.1.0 版开始，确实支持批处理： https : //github.com/Azure/azure-functions-servicebus-extension/issues/15#issuecomment-570682200

Answer 3

I'm just wondering why you need that batch insert. for every single message that triggers your function a new instance of you function is being created by Azure; so performance won't be an issue here. If you still wants to do batching, you may store incoming messages temporarily in a proxy Azure database/data store and query that proxy in specified periods and run your batch insert command. Please keep in mind that you must delete queried records after successful batch insert.

Answer 4

Azure Functions runtime retrieves and processes queue messages in batches. And the default batchSize is 16 and the maximum batchSize is 32..This can be configured for 'queue' triggers by mentioning batchSize in host.json file of the Function App. Refer to know more on host.json

Configuration settings for 'queue' triggers

"queues": {
  "maxPollingInterval": 2000,
  "visibilityTimeout" : "00:00:10",
  "batchSize": 16,
  "maxDequeueCount": 5,
  "newBatchThreshold": 8
}