[英]FIFO / QUEUE with buffer and thread-safe
I'm receiving tons of statistics data that I need to insert into the db. 我收到了大量需要插入数据库的统计数据。 I would like to implement some kind of Queue or FIFO class that keeps all the data and when it reaches to a specific count (buffer), it will send that data to the SQL through bulk insert.
我想实现某种Queue或FIFO类来保存所有数据,当它达到特定计数(缓冲区)时,它会通过批量插入将该数据发送到SQL。 This should be thread-safe.
这应该是线程安全的。
I know how to make the bulk insert. 我知道如何制作批量插入物。 Any advices how to make the queue / list?
有关如何制作队列/列表的建议吗?
Thanks 谢谢
The .net base class library has ConcurrentQueue(Of T)
. .net基类库具有
ConcurrentQueue(Of T)
。 Just import System.Collections.Concurrent
. 只需导入
System.Collections.Concurrent
。
Edit : If you must use a queue, you could create a wrapper class/module that fires off an event when the counter (buffer) reaches a certain amount. 编辑 :如果必须使用队列,则可以创建一个包装类/模块,在计数器(缓冲区)达到一定量时触发事件。
If you don't need strict FIFO I think you should use BlockingCollection
. 如果您不需要严格的FIFO,我认为您应该使用
BlockingCollection
。
It is thread safe and the implementation would look somthing like: 它是线程安全的,实现看起来像:
var collection = new BlockingCollection<Data>();
var sqlinserter = Task.Factory.StartNew(UpdateSql());
while (true) {
Data statistics = FetchStatistics();
if (statistics == null)
break;
collection.Add(statistics);
}
collection.CompleteAdding();
sqlinserter.Wait();
Edit Saw that you wanted to insert a specific count of items in each batch 编辑 Saw您希望在每个批次中插入特定的项目数
void UpdateSql() {
var batch = new List<Data>();
foreach (var item in collection.GetConsumingEnumerable()) {
batch.Add(item);
if (batch.Count > SomeBatchSize) {
InsertIntoSql(batch);
batch.Clear();
}
}
if (batch.Count > 0)
InsertIntoSql(batch); // insert remaining items
}
This is a safe way to deal with it. 这是一种安全的方法来处理它。 Primarily you want to avoid any situation where you can get "stuck" inside a synclock.
主要是你想避免任何你可以在一个synclock内“卡住”的情况。
Public Class TSQueue(Of T)
Private q As New Queue(Of T)
Public Property threshold As Integer = 100
Public Event ThresholdHit As EventHandler(Of EventArgs)
Public Sub EnqueueSafe(value As T)
Dim notify As Boolean = False
SyncLock q
q.Enqueue(value)
If q.Count >= threshold Then
notify = True
End If
End SyncLock
If notify Then
RaiseEvent ThresholdHit(Me, EventArgs.Empty)
End If
End Sub
Public Function DequeueSafe() As T
SyncLock q
Return q.Dequeue()
End SyncLock
End Function
Public Function DequeueAllSafe() As T()
Dim out() As T
SyncLock q
out = q.ToArray()
q.Clear()
End SyncLock
Return out
End Function
Public Function CountSafe() As Integer
SyncLock q
Return q.Count
End SyncLock
End Function
End Class
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.