azure webjob stopped with Unhandled Exception: 503 Server Unavailable

Question

i have a web job with a QueueTrigger. i wrap the process function with try&catch but it seems like azure queue is throwing an unhandled exception that i cant catch in my code. how can i automatically restart the webjob? how can i add retry policy to the webjob sdk?

this is the exception i see in the webjob log (which shows that the error occurs in the webjob sdk timer) :

04/18/2015 11:10:52 > 2d2f34: ERR ] 
[04/18/2015 11:10:52 > 2d2f34: ERR ] Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (503) Server Unavailable. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase`1 cmd, Exception ex)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.<ExistsImpl>b__1d(RESTCommand`1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    --- End of inner exception stack trace ---
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.EndExists(IAsyncResult asyncResult)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass1`1.<CreateCallback>b__0(IAsyncResult ar)
[04/18/2015 11:10:52 > 2d2f34: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.<ExecuteAsync>d__4.MoveNext()
[04/18/2015 11:10:52 > 2d2f34: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext()
[04/18/2015 11:10:52 > 2d2f34: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Threading.ThreadHelper.ThreadStart()
[04/18/2015 11:10:52 > 2d2f34: SYS INFO] Status changed to Failed
[04/18/2015 11:10:52 > 2d2f34: SYS ERR ] Job failed due to exit code -532462766

this is my code:

public static async Task  ManualTrigger([QueueTrigger(ConstantVars.TargetjobQueue)] TargetingJob job, TextWriter log)
{
    try
    {
        AddTextWriterToAppender(log);

        _logger.DebugFormat("Starting job");
        _logger.DebugFormat("Job details: {0}", job);

        await ProcessJob(job).ConfigureAwait(false);

        _logger.Debug("Finished job");

        RemoveTextWriterFromAppender();
    }
    catch (Exception e)
    {
        _logger.Error(string.Format("Unhandled exception was caught in ManualTrigger.\n jobParams={0}",job.ToString()),e);              
    }
}

Answer 1

As you identified, the storage exception is happening outside of your job function so you can't catch it. However, the webjobs SDK will handle failures and retry failed invocations automatically based on it's own internal randomized exponential backoff strategy. You can control the number of retries setting JobHostConfiguration.Queues.MaxDequeueCount on startup.