[英]Azure WebJob logging is crashing host, what can we do?
我们正在使用Azure运行多个WebJob。 我们的WebJob函数之一具有以下签名:
public static Task ProcessFileUploadedMessageAsync(
[QueueTrigger("uploads")] FileUploadedMessage message,
TextWriter logger,
CancellationToken cancellationToken)
此功能正在监视队列中是否有消息指示文件已上传,然后触发文件数据的导入。 请注意,将TextWriter
用作第二个参数:这是由WebJobs API基础结构提供的。
我们的导入过程有点慢(在某些情况下,单个文件的导入可能要花几个小时),因此我们会定期(通过TextWriter
)将消息写入日志以跟踪进度。 不幸的是,由于日志记录异常,我们较大的文件导致WebJob托管过程终止。 这是来自托管进程日志的示例堆栈跟踪:
[04/02/2016 03:44:59 > 660083: ERR ]
[04/02/2016 03:45:00 > 660083: ERR ] Unhandled Exception: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
[04/02/2016 03:45:00 > 660083: ERR ] Parameter name: chunkLength
[04/02/2016 03:45:00 > 660083: ERR ] at System.Text.StringBuilder.ToString()
[04/02/2016 03:45:00 > 660083: ERR ] at System.IO.StringWriter.ToString()
[04/02/2016 03:45:00 > 660083: ERR ] at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ] at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<TryExecuteAsync>d__3.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ] at Microsoft.Azure.WebJobs.Host.Timers.RecurrentTaskSeriesCommand.<ExecuteAsync>d__0.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ] at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ] at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
[04/02/2016 03:45:00 > 660083: ERR ] at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[04/02/2016 03:45:00 > 660083: ERR ] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/02/2016 03:45:00 > 660083: ERR ] at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/02/2016 03:45:00 > 660083: ERR ] at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[04/02/2016 03:45:00 > 660083: ERR ] at System.Threading.ThreadHelper.ThreadStart()
[04/02/2016 03:45:00 > 660083: SYS ERR ] Job failed due to exit code -532462766
[04/02/2016 03:45:01 > 660083: SYS INFO] Process went down, waiting for 0 seconds
[04/02/2016 03:45:01 > 660083: SYS INFO] Status changed to PendingRestart
主要问题在于,此异常不是由我们的代码而是由WebJobs API中的某些东西引发的:
at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()
我们尝试在对TextWriter
的调用周围放置try...catch
块,但这些没有效果。 看来日志消息被缓冲在某个位置,并通过一个单独的线程定期刷新到Azure blob存储。 如果是这种情况,那么我们将无法捕获异常。
是否还有其他人遇到相同的问题,或者有人可以考虑可能的解决方案或解决方法?
为了完整TextWriter
,这是我们使用TextWriter
进行记录的方式:
await logger.WriteLineAsync("some text to log");
没有比这更复杂的了。
更新 :似乎这已在WebJobs SDK GitHub上报告为问题#675 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.