Creating a buffer for Consumer and Producer threads using Queue c# .NET

Question

I am writing a windows service application that is capable of collecting data from sensors like temperature, pressure volume etc...

The frequency at which the data is read is pretty high, there could be a hundred sensors and the data being received could be at a frequency could be one per second per sensor..

I need to store this data to an oracle database, for obvious reasons i dont want to hit the database at such a high rate.

Hence i want to create a Buffer.

My plan is to create a Buffer using the standard .NET Queue, a few threads keep Enqueue data into the queue and another timer driven thread can keep writing into the database at regular intervals.

What i want to know is..?? Is This thread safe If this is not, what is the best way of creating a in memory buffer

Answer 1

To answer your question, as long as you lock accesses, you can have multiple threads access a regular queue.

For me though, I didn't use that and wanted to use queues with locks to keep them thread safe. I have been doing this in c# for one of my programs. I just use a regular queue, and then put a locker on accesses to it (enqueue, dequeue, count). It is completely thread safe if you just lock the accesses.

My setup comes from the tutorial/example here: http://www.albahari.com/threading/part2.aspx#_ProducerConsumerQWaitHandle

My situation is a little different than yours, but pretty similar. For me, my data can come in very fast, and if I don't queue it I lose the data if multiple come in at the same time. Then I have a thread running that slowly takes items off the queue and processes them. This hand-off uses an AutoResetEvent to hold my working-thread until data is ready to be processed. In your case you would use a timer or something that happens regularly.

I copy/pasted my code and tried to change the names. Hopefully I didn't completely break it by missing some name changes, but you should be able to get the gist.

public class MyClass : IDisposable
{
    private Thread sensorProcessingThread = null;
    private Queue<SensorData> sensorQueue = new Queue<SensorData>();
    private readonly object _sensorQueueLocker = new object();
    private EventWaitHandle _whSensorEvent = new AutoResetEvent(false);

    public MyClass () {
        sensorProcessingThread = new Thread(sensorProcessingThread_DoWork);
        sensorProcessingThread.Start();
    }
    public void Dispose()
    {
        // Signal the end by sending 'null'
        EnqueueSensorEvent(null);
        sensorProcessingThread.Join();
        _whSensorEvent.Close();
    }
    // The fast sensor data comes in, locks queue, and then
    // enqueues the data, and releases the EventWaitHandle
    private void EnqueueSensorEvent( SensorData wd )
    {
        lock ( _sensorQueueLocker )
        {
            sensorQueue.Enqueue(wd);
            _whSensorEvent.Set();
        }
    }

    // When asynchronous events come in, I just throw them into queue
    private void OnSensorEvent( object sender, MySensorArgs e )
    {
        EnqueueSensorEvent(new SensorData(sender, e));
    }
    // I have several types of events that can come in,
    // they just get packaged up into the same "SensorData"
    // struct, and I worry about the contents later
    private void FileSystem_Changed( object sender, System.IO.FileSystemEventArgs e )
    {
        EnqueueSensorEvent(new SensorData(sender, e));
    }

    // This is the slower process that waits for new SensorData,
    // and processes it. Note, if it sees 'null' as data,
    // then it knows it should quit the while(true) loop.
    private void sensorProcessingThread_DoWork( object obj )
    {
        while ( true )
        {
            SensorData wd = null;
            lock ( _sensorQueueLocker )
            {
                if ( sensorQueue.Count > 0 )
                {
                    wd = sensorQueue.Dequeue();
                    if ( wd == null )
                    {
                        // Quit the loop, thread finishes
                        return;
                    }
                }
            }
            if ( wd != null )
            {
                try
                {
                    // Call specific handlers for the type of SensorData that was received
                    if ( wd.isSensorDataType1 )
                    {
                        SensorDataType1_handler(wd.sender, wd.SensorDataType1Content);
                    }
                    else
                    {
                        FileSystemChanged_handler(wd.sender, wd.FileSystemChangedContent);
                    }
                }
                catch ( Exception exc )
                {
                    // My sensor processing also has a chance of failing to process completely, so I have a retry
                    // methodology that gives up after 5 attempts
                    if ( wd.NumFailedUpdateAttempts < 5 )
                    {
                        wd.NumFailedUpdateAttempts++;
                        lock ( _sensorQueueLocker )
                        {
                            sensorQueue.Enqueue(wd);
                        }
                    }
                    else
                    {
                        log.Fatal("Can no longer try processing data", exc);
                    }
                }
            }
            else
                _whWatchEvent.WaitOne(); // No more tasks, wait for a signal
        }
    }

Something you could maybe look at is Reactive (Rx) for .net from Microsoft. Check out: https://msdn.microsoft.com/en-us/data/gg577611.aspx and at the bottom of page is a pdf tutorial "Curing the asynchronous blues": http://go.microsoft.com/fwlink/?LinkId=208528 This is something very different but maybe you will see something you like.