Azure Cloud-Service OnStop

Question

I using Azure Cloud Worker Role for processing incoming task from queues. Processing of each task can take up to several hours and each worker-role can handle up to N tasks simultaneously. Basically, it's working.

Now, you can read in documentation that from time to time, the worker role can be shutdown (for software update, OS upgrade, ...). Basically, it's fine. But, this planned shutdown cannot forcedly stop the worker-role already running tasks .

Expected:

When calling the OnStop() method by the environment:

1. the worker role will stop getting new tasks for processing.
2. Wait for running tasks completion.
3. Continue with the planned shutdown.

Actual:

OnStop() method can be block for up to 5 minutes . I cannot guaranty that I'll finish processing the task in 5 minutes - so, this is problem... My task is being killed in the middle of processing and this became unstable situation for my software.

How I'm can avoid this 5 minutes limit? Any tip will be welcome.

Answer 1

How I'm can avoid this 5 minutes limit?

Unfortunately, you can't. This is a hard limit imposed from Azure side. You will need to work around that.

There are two possible solutions I can think of and both of them would require you to rethink about your current architecture:

1. Break your one big task into many smaller tasks and create some kind of work flow.
2. Make your task idempotent so that even if it gets terminated in between (because of worker role shutdown or error in task itself) and when it gets pick up by another instance, it starts again in such a way that your output of the task is not corrupted.

Answer 2

No, you cannot bypass this limit. In general you should not rely on any of your instances running continuously for any long period of time. Instances may be suddenly stopped or they may suddenly disappear (because of an underlying server failure). You software should be designed such that when an instance is restarted (possibly redeployed) or some other instance finds capacity to take a previously released work item that work item is reprocessed without any adverse effects.