FIFO / QUEUE with buffer and thread-safe

Question

I'm receiving tons of statistics data that I need to insert into the db. I would like to implement some kind of Queue or FIFO class that keeps all the data and when it reaches to a specific count (buffer), it will send that data to the SQL through bulk insert. This should be thread-safe.

I know how to make the bulk insert. Any advices how to make the queue / list?

Thanks

Answer 1

The .net base class library has ConcurrentQueue(Of T) . Just import System.Collections.Concurrent .

Edit : If you must use a queue, you could create a wrapper class/module that fires off an event when the counter (buffer) reaches a certain amount.

Answer 2

If you don't need strict FIFO I think you should use BlockingCollection .

It is thread safe and the implementation would look somthing like:

var collection = new BlockingCollection<Data>();

var sqlinserter = Task.Factory.StartNew(UpdateSql());

while (true) {
    Data statistics = FetchStatistics();
    if (statistics == null)
        break;
    collection.Add(statistics);
}
collection.CompleteAdding();
sqlinserter.Wait();

Edit Saw that you wanted to insert a specific count of items in each batch

void UpdateSql() {
    var batch = new List<Data>();
    foreach (var item in collection.GetConsumingEnumerable()) {
        batch.Add(item);
        if (batch.Count > SomeBatchSize) {
            InsertIntoSql(batch);
            batch.Clear();
        }
    }
    if (batch.Count > 0)
        InsertIntoSql(batch); // insert remaining items
}

Answer 3

This is a safe way to deal with it. Primarily you want to avoid any situation where you can get "stuck" inside a synclock.

Public Class TSQueue(Of T)

    Private q As New Queue(Of T)
    Public Property threshold As Integer = 100
    Public Event ThresholdHit As EventHandler(Of EventArgs)

    Public Sub EnqueueSafe(value As T)
        Dim notify As Boolean = False
        SyncLock q
            q.Enqueue(value)
            If q.Count >= threshold Then
                notify = True
            End If
        End SyncLock
        If notify Then
            RaiseEvent ThresholdHit(Me, EventArgs.Empty)
        End If
    End Sub

    Public Function DequeueSafe() As T
        SyncLock q
            Return q.Dequeue()
        End SyncLock
    End Function

    Public Function DequeueAllSafe() As T()
        Dim out() As T
        SyncLock q
            out = q.ToArray()
            q.Clear()
        End SyncLock
        Return out
    End Function

    Public Function CountSafe() As Integer
        SyncLock q
            Return q.Count
        End SyncLock
    End Function

End Class