ExecuteNonQuery hangs forever in .NET 4.6.1

Question

There is a rare situation when process just freezes during execution of ExecuteNonQuery :

cmd.ExecuteNonQuery();

CommandTimeout is set to 0 (no timeout) and a simple UPDATE statement is being executed. This works usually fast and the issue can be reproduced in average once per week. But when it happens, it's a complete freeze/hung for the process. Last time it was reproduced we used ProcDump to get a dump file for that particular process and can see this stacktrace there. The only difference between other SO questions are first few lines (deadlock?)

OS Thread Id: 0x4168 (34)
Current frame: ntdll!NtWaitForSingleObject+0xc
ChildEBP RetAddr  Caller, Callee
23eed994 7587f699 KERNELBASE!WaitForSingleObjectEx+0x99, calling ntdll!NtWaitForSingleObject
23eeda08 7587f5f2 KERNELBASE!WaitForSingleObject+0x12, calling KERNELBASE!WaitForSingleObjectEx
23eeda1c 5829e5e2 System_Data!Np::ReadSync+0x205, calling KERNELBASE!WaitForSingleObject
23eeda58 58293812 System_Data!SNIReadSync+0x64
23eeda88 5828a795 System_Data!SNIReadSyncOverAsync+0x25, calling System_Data!SNIReadSync
23eedaa4 53ce748e (MethodDesc 5395aa60 +0x46 DomainBoundILStubClass.IL_STUB_PInvoke(SNI_ConnWrapper*, SNI_Packet**, Int32))
23eedac8 53ce748e (MethodDesc 5395aa60 +0x46 DomainBoundILStubClass.IL_STUB_PInvoke(SNI_ConnWrapper*, SNI_Packet**, Int32))
23eedae4 53cf3c57 (MethodDesc 539498d0 +0x53 SNINativeMethodWrapper.SNIReadSyncOverAsync(System.Runtime.InteropServices.SafeHandle, IntPtr ByRef, Int32)), calling 08a34a78
23eedb04 53e00e5e (MethodDesc 5394a534 +0x5e System.Data.SqlClient.TdsParserStateObject.GetTimeoutRemaining()), calling (MethodDesc 53950710 +0 System.Data.SqlClient.TdsParserStaticMethods.GetTimeoutMilliseconds(Int64))
23eedb18 53e027fb (MethodDesc 5394a90c +0x73 System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()), calling (MethodDesc 539498d0 +0 SNINativeMethodWrapper.SNIReadSyncOverAsync(System.Runtime.InteropServices.SafeHandle, IntPtr ByRef, Int32))
23eedb4c 53e02726 (MethodDesc 5394a8f8 +0x6e System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()), calling (MethodDesc 5394a90c +0 System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync())
23eedb5c 53e0181b (MethodDesc 5394a624 +0x63 System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()), calling (MethodDesc 5394a8f8 +0 System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket())
23eedb68 53e019b3 (MethodDesc 5394a660 +0x2b System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte ByRef)), calling (MethodDesc 5394a624 +0 System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer())
23eedb78 53dedda0 (MethodDesc 5394f064 +0x20c System.Data.SqlClient.TdsParser.TryRun(System.Data.SqlClient.RunBehavior, System.Data.SqlClient.SqlCommand, System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.BulkCopySimpleResultSet, System.Data.SqlClient.TdsParserStateObject, Boolean ByRef)), calling (MethodDesc 5394a660 +0 System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte ByRef))
23eedc0c 53d1d4f0 (MethodDesc 538fe598 +0x7c System.Data.SqlClient.SqlCommand.FinishExecuteReader(System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.RunBehavior, System.String, Boolean, Boolean, Boolean)), calling (MethodDesc 5394f064 +0 System.Data.SqlClient.TdsParser.TryRun(System.Data.SqlClient.RunBehavior, System.Data.SqlClient.SqlCommand, System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.BulkCopySimpleResultSet, System.Data.SqlClient.TdsParserStateObject, Boolean ByRef))
23eedc50 53d1d00b (MethodDesc 538fe56c +0xa2f System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, Boolean, Int32, System.Threading.Tasks.Task ByRef, Boolean, Boolean, System.Data.SqlClient.SqlDataReader, Boolean)), calling (MethodDesc 538fe598 +0 System.Data.SqlClient.SqlCommand.FinishExecuteReader(System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.RunBehavior, System.String, Boolean, Boolean, Boolean))
23eedc68 746b59db clr!JIT_Security_Prolog_Framed+0x15c, calling clr!_EH_epilog3
23eedd18 53d1c27d (MethodDesc 538fe53c +0x605 System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String, System.Threading.Tasks.TaskCompletionSource`1<System.Object>, Int32, System.Threading.Tasks.Task ByRef, Boolean ByRef, Boolean, Boolean)), calling 08a34c1c
23eedd40 72017e17 (MethodDesc 71da0878 +0x37 System.Collections.Hashtable.get_Item(System.Object))
23eedda8 53d16fa3 (MethodDesc 538fe22c +0x187 System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(System.Threading.Tasks.TaskCompletionSource`1<System.Object>, System.String, Boolean, Int32, Boolean ByRef, Boolean, Boolean)), calling 08a34b9c
23eede04 53d15f98 (MethodDesc 538fe160 +0xd0 System.Data.SqlClient.SqlCommand.ExecuteNonQuery()), calling 08a38678

---

I have found several SO questions regarding this topic as well, but none of the answers seems legit.

1. Need help diagnosing SQL Server strange query timeouts from C#
2. SqlDataReader hangs on GetValue() method and SNIReadSyncOverAsync
3. ADO.Net SQLCommand.ExecuteReader() slows down or hangs
4. SqlDataReader.GetValue Hangs
5. Windbg - process frozen on SNIReadSyncOverAsync but no deadlock found
6. https://social.microsoft.com/Forums/en-US/3f62d9ba-87e8-446d-b8ba-a3dfdf78a8e9/no-connection-could-be-made-because-the-target-machine-actively-refused-the-connection?forum=sharepointadmin

Answer 1

I think the problem is clearly indicated in the last frame of the stack-trace: WaitForSingleObjectEx

Waits until the specified object is in the signaled state, an I/O completion routine or asynchronous procedure call (APC) is queued to the thread, or the time-out interval elapses.

...

If the timeout dwMilliseconds is INFINITE , the function will return only when the object is signaled or an I/O completion routine or APC is queued.

...

A thread that uses a wait function with no time-out interval may cause the system to become deadlocked .

So, yes, I believe it is a deadlock and I would recommend to never use an infinite time-out; use a sensible time-out value in a loop of multiple retries (5 should be enough) always waiting a random amount of time between tries. Note the random wait is important to resolve deadlocks otherwise the deadlock victim will confront the next item in the loop exactly after the same amount of time and, as the operation potentially takes the same amount of time, it will easily become the deadlock victim again.