繁体   English   中英

Azure WebJob日志记录使主机崩溃,我们该怎么办?

[英]Azure WebJob logging is crashing host, what can we do?

我们正在使用Azure运行多个WebJob。 我们的WebJob函数之一具有以下签名:

public static Task ProcessFileUploadedMessageAsync(
    [QueueTrigger("uploads")] FileUploadedMessage message,
    TextWriter logger,
    CancellationToken cancellationToken)

此功能正在监视队列中是否有消息指示文件已上传,然后触发文件数据的导入。 请注意,将TextWriter用作第二个参数:这是由WebJobs API基础结构提供的。

我们的导入过程有点慢(在某些情况下,单个文件的导入可能要花几个小时),因此我们会定期(通过TextWriter )将消息写入日志以跟踪进度。 不幸的是,由于日志记录异常,我们较大的文件导致WebJob托管过程终止。 这是来自托管进程日志的示例堆栈跟踪:

[04/02/2016 03:44:59 > 660083: ERR ] 
[04/02/2016 03:45:00 > 660083: ERR ] Unhandled Exception: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
[04/02/2016 03:45:00 > 660083: ERR ] Parameter name: chunkLength
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Text.StringBuilder.ToString()
[04/02/2016 03:45:00 > 660083: ERR ]    at System.IO.StringWriter.ToString()
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<TryExecuteAsync>d__3.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.RecurrentTaskSeriesCommand.<ExecuteAsync>d__0.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ThreadHelper.ThreadStart()
[04/02/2016 03:45:00 > 660083: SYS ERR ] Job failed due to exit code -532462766
[04/02/2016 03:45:01 > 660083: SYS INFO] Process went down, waiting for 0 seconds
[04/02/2016 03:45:01 > 660083: SYS INFO] Status changed to PendingRestart

主要问题在于,此异常不是由我们的代码而是由WebJobs API中的某些东西引发的:

at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()

我们尝试在对TextWriter的调用周围放置try...catch块,但这些没有效果。 看来日志消息被缓冲在某个位置,并通过一个单独的线程定期刷新到Azure blob存储。 如果是这种情况,那么我们将无法捕获异常。

是否还有其他人遇到相同的问题,或者有人可以考虑可能的解决方案或解决方法?


为了完整TextWriter ,这是我们使用TextWriter进行记录的方式:

await logger.WriteLineAsync("some text to log");

没有比这更复杂的了。


更新 :似乎这已在WebJobs SDK GitHub上报告为问题#675

您应该向方法添加async ,如下所示:

public async static Task ProcessFileUploadedMessageAsync(...)

要获取详细信息,您可以查看本文

暂无
暂无

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM