azure queue storage - putting a message back in the queue

Question

Im building a queue processing webjob to read messages from a queue and use the data to retrieve a report uri from blob storage then send this as a link in an email. I have the process working perfectly well, but I need to send the email within a specific time window.

I have another process (webjob) that retrieves this data from a sql backend and places it in the email queue.

This webjob runs every 30 mins and only gets data for the current day and within a 2 hour window of the current time. So I know that anything in the queue is for today and within 2 hours of 'now'. How would i narrow this down further to read data from the queue, and if the 'email out' time is set to say 19:00 and the current time is 18:00, I can put this message back in the queue to read it again later, next time it should be closer to 19:00 then I can process it and send it out in an email.
The time doesn't have to be spot on, so even if its within, say 30 mins of 19:00 (or whatever time its set to be sent) It can be processed. So Im effectively taking an object from the queue, checking its time and if its not within say 30 mins of its allotted 'email out' time, I place it back in the queue and its processed again

** in my webjob, I have a 'Functions' class that contains a method 'ProcessQueueMessage' which is triggered whenever a message is placed in the queue.

// This function will get triggered/executed when a new message is written 
    // on an Azure Queue called queue.
    public async Task ProcessQueueMessage([QueueTrigger("%reportgenerator%")] Data.Dto.Schedule.ScheduleDto schedule)
    {
        var reports = await this._scheduledReportGenerationService.GenerateScheduledEmails(schedule.ID);
    }

The ScheduleDto class will have a generation time property, I can read this and compare it to the current time and process it only if its within my specified 'time window'. How would i stop the queue message from being deleted here so that I can re-process it ?

Answer 1

When you add the message to the queue just set the initialVisibilityDelay so that the message isn't seen until the minimum process time.

CloudQueue queue = queueClient.GetQueueReference(queueName);
var msg = new CloudQueueMessage("Hello World!");
TimeSpan timeSpanDelay = GetEarliestProcessTime();
await queue.AddMessageAsync(msg, null, timeSpanDelay, null, null);

CloudQueue.AddMessage

Answer 2

So there're a few things with Azure Storage Queues that is going to help you with this scenario:

Regarding putting the message back in the queue, you don't have to do anything special. It is a feature offered by Storage Queues. When you dequeue a message ( GET Message in Azure Storage lingo), the message becomes invisible for certain amount of time and if not deleted by the process which dequeued it will become visible again and can be picked up by another process.

So when you dequeue the message, check the time and if it is not the right time you don't do anything. However please make sure that once the message is processed, you delete that message otherwise it will be picked up again.

One more thing you could do is when you dequeue the message and find that it is not the right time to process that message, you update that message and set its visibility timeout property to a value that will make the message visible again closer to the processing time. For example, you dequeued the message at 18:00 and found that this message needs to processed at 19:00. In this case you will update the message and set its visibility timeout to 50 minutes (or a value greater than 30 minutes as 30 minutes is the schedule for your web job). What this will do is ensure that when your webjob runs at 18:30, this message will not be picked up by the web job because the message will only become visible at 18:50.

You can read more about updating message here: https://docs.microsoft.com/en-us/rest/api/storageservices/update-message and about dequeuing messages here: https://docs.microsoft.com/en-us/rest/api/storageservices/get-messages .

Update

I completely forgot that it is in a WebJob so doing nothing would actually delete the message. I guess you have 2 options (kind of repeating what's mentioned in the comments):

1. Throw an exception instead of doing nothing. This will ensure that WebJob processor does not delete the message. I have not tried it myself but you can also update the message and set its visibility timeout to a value closer to desired time in the WebJob itself (and then throw an exception). This is kind of an anti-pattern though.
2. You add a new message in the queue and set its initial visibility timeout to a value closer to desired time (this is also covered in another answer) and let this message delete.

Answer 3

When you en-queue an item in Azure Storage queues you can add extra detail to the item that will cause the item to be hidden for a configurable duration. If your batch job can only run every two hours but you want to delay sending emails with more fine-grained time control then I suggest the two-hourly en-queuing batch job could use this "initialVisibilityDelay" feature.

Here is another SO question that describes the API.

Azure storage queue message (show at specific time)