SQL Server与Azure Eventhub + Logic应用程序并行插入

Question

I'm using Azure Event Hub to stream Azure diagnostics data to a Logic app to save it into an Azure SQL table for monitoring purposes. This works great. However, it sometimes occurs that EventHub sends duplicates. To avoid duplicate inserts I'm using a INSERT INTO statement with a WHERE NOT EXISTS clause. However, very infrequently I am still getting double rows. The Logic app runs parallel so I guess this is causing the issue. I think sometimes it does the same insert on exactly the same time, which causes the WHERE NOT EXISTS clause not to work.

Does anyone know a workaround? i'd rather not do a DELETE from and remove duplicate rows afterwards as I want to put a unique key constraint on the table.

Answer 1

You have to have some idempotency checking in you function. When scaling happens the new scale unit will take over processing a partition and thus reprocess the same messages.

I don't know if Service Bus can queue those messages. I advise you to report to the Event Hubs team.