Hangfire - Win32Exception (258): 等待操作超时

Question

执行作业时，有时会出现以下异常：

System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
 ---> System.ComponentModel.Win32Exception (258): The wait operation timed out.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite, String methodName)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader)
   at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command)
   at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)
   at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
   at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple)
   at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstance instance, Func`2 action, String message)
   at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message)
   at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)
ClientConnectionId:cf3220e9-bbf5-40f2-8dc5-0d7836736771
Error Number:-2,State:0,Class:11

这是失败的作业方法：

[AutomaticRetry(Attempts = 0)]
[DisableConcurrentExecution(30 * 60)]
public async System.Threading.Tasks.Task ExecuteAsync(int id, PerformContext context, CancellationToken cancellationToken)
{
    ...
}

由于DisableConcurrentExecutionAttribute的超时时间已经设置为 30 分钟，这里有什么问题？ 我什至将 hangfire sql 服务器配置中的CommandTimeout增加到 60 秒：

builder.Services.AddHangfire((provider, config) => config
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseSqlServerStorage(databaseConnectionStr, new SqlServerStorageOptions
    {
        CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
        CommandTimeout = TimeSpan.FromSeconds(60),
        SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
        QueuePollInterval = TimeSpan.Zero,
        UseRecommendedIsolationLevel = true,
        DisableGlobalLocks = true,
        PrepareSchemaIfNecessary = false
    })
    .WithJobExpirationTimeout(TimeSpan.FromDays(30 * 6))
    .UseFilter(provider.GetRequiredService<SendEMailOnFailureFilter>())
    .UseConsole());

如果我手动重新排队相同的作业，它每次都有效。

更新：

这是一个失败作业的屏幕截图，我们可以看到它在总共 37 秒后就已经失败了，这小于DisableConcurrentExecution的超时和 hangfire 中配置的CommandTimeout (SQL)。 在另一个实例中，它仅在 27 秒后就失败了。

更新 2：

正如@jaroslav 所建议的，我尝试了一个具有 70 秒睡眠/延迟的虚拟作业，但它确实完美地工作了。 甚至DisableConcurrentExecution属性也运行良好：

[AutomaticRetry(Attempts = 0)]
[DisableConcurrentExecution(30 * 60)]
public override async Task ExecuteAsync(PerformContext context, CancellationToken cancellationToken)
{
    await Task.Delay(70 * 1000, cancellationToken);
}

Answer 1

我怀疑 hangfire 在执行任务的同时在做一些其他的工作。 这超过了默认的sqlCommandTimeout （30 秒）。 尝试增加默认的 SqlCommand 超时：

using (SqlCommand cmd = new SqlCommand(query, con))
{
    cmd.CommandType = CommandType.StoredProcedure;
    cmd.CommandTimeout = 3600; // 1 hour
    await cmd.ExecuteNonQueryAsync();
}