Find Reason for The wait operation timed out

Question

I have below code to get data from database. It hosted in WCF Service (IIS Server).

public DataTable GetDocument(int DocumentID)
{
    SqlCommand sqlCommand = null;
    SqlConnection sqlConnection = null;
    SqlDataAdapter sqlDataAdapter = null;
    DataTable dataTable = null;

    try
    {
        sqlConnection = new SqlConnection();
        sqlConnection.ConnectionString = "Connection_String";

        sqlCommand = new SqlCommand();
        sqlCommand.CommandText = "dbo.[Get_Document]";
        sqlCommand.CommandType = CommandType.StoredProcedure;
        sqlCommand.Connection = sqlConnection;
        sqlCommand.Parameters.Add(new SqlParameter("@document_id", SqlDbType.BigInt, 8, ParameterDirection.Input, true, 19, 0, "", DataRowVersion.Proposed, DocumentID));

        sqlConnection.Open();

        sqlDataAdapter = new SqlDataAdapter(sqlCommand);
        dataTable = new DataTable("Document");
        sqlDataAdapter.Fill(dataTable);

        return dataTable;
    }
    catch (Exception ex)
    {
        ErrorLog.LogError(ex, "DocumentID = " + DocumentID);
    }
    finally
    {
        if (sqlConnection != null)
        {
            if(sqlConnection.State != ConnectionState.Closed)
                sqlConnection.Close();

            sqlConnection.Dispose();
            sqlConnection = null;
        }

        if (sqlCommand != null)
        {
            sqlCommand.Dispose();
            sqlCommand = null;
        }

        if (dataTable != null)
        {
            dataTable.Dispose();
            dataTable = null;
        }

        if (sqlDataAdapter != null)
        {
            sqlDataAdapter.Dispose();
            sqlDataAdapter = null;
        }
    }

    return null;
}

It is working correctly but some time randomly we got the below timeout exception.

PARAMETER : DocumentID =  987456

EXCEPTION

Error Message: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

Error Source: .Net SqlClient Data Provider

Error Stack Trace:    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)
   at System.Data.Common.DbDataAdapter.Fill(DataTable[] dataTables, Int32 startRecord, Int32 maxRecords, IDbCommand command, CommandBehavior behavior)
   at System.Data.Common.DbDataAdapter.Fill(DataTable dataTable)
   at DAL.DocumentController.GetDocument(int DocumentID)

TargetSite : Void OnError(System.Data.SqlClient.SqlException, Boolean, System.Action`1[System.Action])


INNER EXCEPTION

Error Message: The wait operation timed out


BASE EXCEPTION  

Error Message: The wait operation timed out

When timeout exception is occurred at that time we logged exception with appropriate stored procedure's parameter. later on when we investigating the exception logs at that time we perform same operation with same parameter but at that time system behaves as expected (not able to replicate exception even with same parameter).

As far as I know many causes involved for timeout exception. We have already set enough timeout so we don't want to increase timeout in our application.

We just want to know the reason of timeout exception why it is occurred in particular time frame (eg It is because of any deadlock, connectivity issue etc.) and logged that reason so later on we can investigate in correct direction.

Is there any way to know the reason of timeout exception?

Thanks.

Answer 1

I think you are right. The timeout exception might due to the client request amount overtake the maximum server load or resource deadlock. Whether WCF or SQL server there is a concurrent request limitation. Here is a template to configure this.

<behaviors>
      <serviceBehaviors>
        <behavior>
          <serviceThrottling maxConcurrentCalls="500" maxConcurrentSessions="10000" maxConcurrentInstances="100" />
        </behavior>
      </serviceBehaviors>

The instancing mode and the concurrency mode have an impact on the way of processing the client's request.
https://docs.microsoft.com/en-us/dotnet/framework/wcf/feature-details/sessions-instancing-and-concurrency
https://docs.microsoft.com/en-us/dotnet/api/system.servicemodel.servicebehaviorattribute.concurrencymode?view=netframework-4.8
The combination of instancing mode and concurrency mode might cause a deadlock so that results in the timeout exception.
Finally, the database connection string had better a separate username/password, because the process of IIS hosting the WCF application, it will be replaced with the IIS application pool identity.