Hangfire - Win32Exception (258): 等待操作超時

Question

執行作業時，有時會出現以下異常：

System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
 ---> System.ComponentModel.Win32Exception (258): The wait operation timed out.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite, String methodName)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader)
   at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command)
   at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)
   at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
   at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple)
   at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstance instance, Func`2 action, String message)
   at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message)
   at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)
ClientConnectionId:cf3220e9-bbf5-40f2-8dc5-0d7836736771
Error Number:-2,State:0,Class:11

這是失敗的作業方法：

[AutomaticRetry(Attempts = 0)]
[DisableConcurrentExecution(30 * 60)]
public async System.Threading.Tasks.Task ExecuteAsync(int id, PerformContext context, CancellationToken cancellationToken)
{
    ...
}

由於DisableConcurrentExecutionAttribute的超時時間已經設置為 30 分鍾，這里有什么問題？ 我什至將 hangfire sql 服務器配置中的CommandTimeout增加到 60 秒：

builder.Services.AddHangfire((provider, config) => config
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseSqlServerStorage(databaseConnectionStr, new SqlServerStorageOptions
    {
        CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
        CommandTimeout = TimeSpan.FromSeconds(60),
        SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
        QueuePollInterval = TimeSpan.Zero,
        UseRecommendedIsolationLevel = true,
        DisableGlobalLocks = true,
        PrepareSchemaIfNecessary = false
    })
    .WithJobExpirationTimeout(TimeSpan.FromDays(30 * 6))
    .UseFilter(provider.GetRequiredService<SendEMailOnFailureFilter>())
    .UseConsole());

如果我手動重新排隊相同的作業，它每次都有效。

更新：

這是一個失敗作業的屏幕截圖，我們可以看到它在總共 37 秒后就已經失敗了，這小於DisableConcurrentExecution的超時和 hangfire 中配置的CommandTimeout (SQL)。 在另一個實例中，它僅在 27 秒后就失敗了。

更新 2：

正如@jaroslav 所建議的，我嘗試了一個具有 70 秒睡眠/延遲的虛擬作業，但它確實完美地工作了。 甚至DisableConcurrentExecution屬性也運行良好：

[AutomaticRetry(Attempts = 0)]
[DisableConcurrentExecution(30 * 60)]
public override async Task ExecuteAsync(PerformContext context, CancellationToken cancellationToken)
{
    await Task.Delay(70 * 1000, cancellationToken);
}

Answer 1

我懷疑 hangfire 在執行任務的同時在做一些其他的工作。 這超過了默認的sqlCommandTimeout （30 秒）。 嘗試增加默認的 SqlCommand 超時：

using (SqlCommand cmd = new SqlCommand(query, con))
{
    cmd.CommandType = CommandType.StoredProcedure;
    cmd.CommandTimeout = 3600; // 1 hour
    await cmd.ExecuteNonQueryAsync();
}