如何查看Azure Event Hub中的最新数据？ 。净

Question

I have a problem about how to peek recent data from Event Hub by .NET application. Our demand is to continuously read/get the last 1-day data in Event Hub, every hour. For example, at 6 o'clock, I want to get the data from 6 o'clock yesterday to 6 o'clock today. Then, at 7 o'clock, I want to get the data from 7 o'clock yesterday to 7 o'clock today.

I have tried to receive events from Azure Event Hubs, following the tutorial . But it doesn't satisify my demand. My comprehension to this receiving events process is that, every time there is a new event coming to Event Hub, a signal will be set up and EventProcessorHost class will be trigged to get the event data. ( I doubt whether my comprehension is right. ) However, in this method, a event data can only be accessed once. It cannot be accessed in the next receiving operations because it disappers in Event Hub.

Is there a method achieving the above demand?

Meanwhile, I also want to know how to use " offset " in receiving events process. I know its concept, but it is better there is a demo of how to use it.

I will be appriciate if you have any advice. :)

Answer 1

This is not a scenario where event hub is designed for. It is designed to handle incoming data at scale and the typical use case is to process that data as fast as possible.

In your case you want to process largely the same data over and over again, with an hour intervals. The EventProcessrHost is meant to be running continuously in a background process.

It might be a lot easier to store incoming data to blob storage in a format that includes the hour component like container\\date\\time\\blob1.json (\\container\\2019-12-22\\07\\blob1.json) and have, for example, a schedule triggered azure function that than knows what blobs to process based on the time it is triggered.

My comprehension to this receiving events process is that, every time there is a new event coming to Event Hub, a signal will be set up and EventProcessorHost class will be trigged to get the event data. (I doubt whether my comprehension is right.) However, in this method, a event data can only be accessed once. It cannot be accessed in the next receiving operations because it disappers in Event Hub.

That is correct, the EventProcessorHost will run on new events. Events are processed once using the concept of checkpointing. You can play with offsets to rewind the stream but I think my alternative is much easier.

Another technology that might be useful to you and plays well with event hub is Azure Stream Analytics , it allows you to define time based windows but 24 hrs might be a bit to much for this, not sure about that.