Best way to receive high volume tcp socket financial data from a feeder in c#?

Question

I am developing a C# Windows Service that will receive financial quotations from a feeder that uses TCP. My project must receive and process a large volume of data, for I will be tracking 140 different assets that are used to update an SQL database every second.

I am using a looping to pool data from the socket, in a BackgroundWork thread:

  try
  {
    // Send START command with the assets.
    if (!AeSocket.AeSocket.Send(Encoding.ASCII.GetBytes(String.Format("{0}{1}START{1}BC|ATIVO{1}{2}{3}", GlobalData.GlobalData.Id, GlobalData.GlobalData.Tab, AeSocket.AeSocket.GetAssets(),
                                                                      GlobalData.GlobalData.Ret))).Contains("OK"))
    {
      throw new Exception("Advise was no accepted.");
    }

    // Pool the socket and send all received string to the queue for processing in the background.
    while (true)
    {
      // Make sure the connection to the socket is still active.
      if (!AeSocket.AeSocket.Client.Connected)
      {
        throw new Exception("The connection was closed.");
      }

      // If no data is available in the socket, loop and keep waiting.
      if (!AeSocket.AeSocket.Client.Poll(-1, SelectMode.SelectRead))
      {
        continue;
      }

      // There are data waiting to be read.
      var data = new Byte[AeSocket.AeSocket.ReadBufferSize];
      var bytes = AeSocket.AeSocket.Client.Receive(data, 0);
      AeSocket.AeSocket.Response = Encoding.Default.GetString(data, 0, bytes);

      // Push into the queue for further processing in a different thread.
      GlobalData.GlobalData.RxQueue.Add(AeSocket.AeSocket.Response);
    }
  }
  catch
  {
    backgroundWorkerMain.CancelAsync();
  }
  finally
  {
    AeSocket.AeSocket.Client.Close();
    AeSocket.AeSocket.Client.Dispose();
  }

The received data is being processed in a separated thread to avoid blocking the socket receiving activity, due to the high data volume received. I am using a BlockingCollection (RxQueue).

This collection is being observed as shown in the following code snippet:

  // Subscribe to the queue for string processing in another thread.
  // This is a blocking observable queue, so it is run in this background worker thread.
  GlobalData.GlobalData.Disposable = GlobalData.GlobalData.RxQueue
    .GetConsumingEnumerable()
    .ToObservable()
    .Subscribe(c =>
    {
      try
      {
        ProcessCommand(c);
      }
      catch
      {
        // Any error will stop the processing.
        backgroundWorkerMain.CancelAsync();
      }
    });

The data is then added to a ConcurrentDictionary, to be read asynchronously by an one sec timer and saved to an SQL database:

      // Add or update the quotation record in the dictionary.
      GlobalData.GlobalData.QuotDict.AddOrUpdate(dataArray[0], quot, (k, v) => quot);

The one second system timer is processed in another BackgroundWorker thread and it saves the quotations data read from the ConcurrentDictionary.

  // Update the gridViewAssetQuotations.
  var list = new BindingList<MyAssetRecord>();

  foreach (var kv in GlobalData.GlobalData.QuotDict)
  {
    // Add the data in a database table...
  }

Is this a good approach to this situation?

Using a BlockingCollection as an assynchronous queue and a ConcurrentDictionary to allow asynchronous reading from another thread is a good way of doing this?

What about the way I used to pool data from the socket:

      // If no data is available in the socket, loop and keep waiting.
      if (!AeSocket.AeSocket.Client.Poll(-1, SelectMode.SelectRead))
      {
        continue;
      }

Is there a better way of doing it?

Also, I have to send a KeepAlive command every 4 seconds to the TCP server. Could I do it totally asynchronously, ignoring the pooling loop above, using another system timer, or it must be synchronized to the pooling operation? The server only allows connection on one port.

Thanks in advance for any suggestions.

Eduardo Quintana

Answer 1

Your loop will lead to high processor usage. Put a sleep (say 1 ms) in case there is no data to process, increase the sleep time if there is still no data in the next cycle, decrease the sleep time if you get data. That way the reader loop will auto adjust to data traffic and would save on processor cycles. Rest all looks good.