Azure WebJob logging is crashing host, what can we do?

Question

We are using Azure to run several WebJobs. One of our WebJob functions has the following signature:

public static Task ProcessFileUploadedMessageAsync(
    [QueueTrigger("uploads")] FileUploadedMessage message,
    TextWriter logger,
    CancellationToken cancellationToken)

This function is monitoring a queue for a message that indicates a file has been uploaded, which then triggers an import of the file's data. Note the use of a TextWriter as the second argument: this is supplied by the WebJobs API infrastructure.

Our import process is kind of slow (can be several hours for a single file import in some cases), so we periodically write messages to the log (via the TextWriter ) to track our progress. Unfortunately, our larger files are causing the WebJob hosting process to be terminated due to a logging exception. Here is a sample stack trace from the hosting process log:

[04/02/2016 03:44:59 > 660083: ERR ] 
[04/02/2016 03:45:00 > 660083: ERR ] Unhandled Exception: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
[04/02/2016 03:45:00 > 660083: ERR ] Parameter name: chunkLength
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Text.StringBuilder.ToString()
[04/02/2016 03:45:00 > 660083: ERR ]    at System.IO.StringWriter.ToString()
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<TryExecuteAsync>d__3.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.RecurrentTaskSeriesCommand.<ExecuteAsync>d__0.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ThreadHelper.ThreadStart()
[04/02/2016 03:45:00 > 660083: SYS ERR ] Job failed due to exit code -532462766
[04/02/2016 03:45:01 > 660083: SYS INFO] Process went down, waiting for 0 seconds
[04/02/2016 03:45:01 > 660083: SYS INFO] Status changed to PendingRestart

The main problem is that this exception is being thrown not by our code but by something in the WebJobs API:

at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()

We tried putting try...catch blocks around our calls to the TextWriter but these had no effect. It would appear that log messages are buffered somewhere and periodically flushed to Azure blob storage by a separate thread. If this is the case, then it would follow that we have no way of trapping the exception.

Has anyone else come across the same problem, or can anyone think of a possible solution or workaround?

---

For completeness, here is how we are using the TextWriter for logging:

await logger.WriteLineAsync("some text to log");

Nothing more complicated than that.

---

UPDATE : Seems as though this has been reported as issue #675 on the WebJobs SDK GitHub .

Answer 1

You should add async to method, just like this:

public async static Task ProcessFileUploadedMessageAsync(...)

To get details, you can see this article