[英]C# BlockingCollection loop on IsCompleted instead of GetConsumingEnumerable
In an application, many actions need to be logged in the database.在应用程序中,需要将许多操作记录在数据库中。 but this logging process should not slow down the request.
但是这个记录过程不应该减慢请求。 So they should be done in an asynchronous queue or something.
所以它们应该在异步队列或其他东西中完成。
This is my big picture:这是我的大图:
And this is my example implementation:这是我的示例实现:
Model:模型:
public class ActivityLog
{
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public string Id { get; set; }
public IPAddress IPAddress { get; set; }
public string Action { get; set; }
public string? Metadata { get; set; }
public string? UserId { get; set; }
public DateTime CreationTime { get; set; }
}
Queue:队列:
public class LogQueue
{
private const int QueueCapacity = 1_000_000; // is one million enough?
private readonly BlockingCollection<ActivityLog> logs = new(QueueCapacity);
public bool IsCompleted => logs.IsCompleted;
public void Add(ActivityLog log) => logs.Add(log);
public IEnumerable<ActivityLog> GetConsumingEnumerable() => logs.GetConsumingEnumerable();
public void Complete() => logs.CompleteAdding();
}
Worker (the problem is here) :工人(问题就在这里) :
public class DbLogWorker : IHostedService
{
private readonly LogQueue queue;
private readonly IServiceScopeFactory scf;
private Task jobTask;
public DbLogWorker(LogQueue queue, IServiceScopeFactory scf)
{
this.queue = queue;
this.scf = scf;
jobTask = new Task(Job, TaskCreationOptions.LongRunning);
}
private void Job()
{
using var scope = scf.CreateScope();
var dbContext = scope.ServiceProvider.GetRequiredService<ApplicationDbContext>();
// The following code does not work
// My intension was to reduce DB trips
//while(!queue.IsCompleted)
//{
// var items = queue.GetConsumingEnumerable();
// dbContext.AddRange(items);
// dbContext.SaveChanges();
//}
// But this works
// If I have 10 items available, I'll have 10 DB trips (not good), right?
foreach (var item in queue.GetConsumingEnumerable())
{
dbContext.Add(item);
dbContext.SaveChanges();
}
}
public Task StartAsync(CancellationToken cancellationToken)
{
jobTask.Start();
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
queue.Complete();
jobTask.Wait(); // or 'await jobTask' ?
return Task.CompletedTask; // unnecessary if I use 'await jobTask'
}
}
Dependency Injection:依赖注入:
builder.Services.AddSingleton<LogQueue>();
builder.Services.AddHostedService<DbLogWorker>();
Controller:控制器:
[HttpGet("/")]
public IActionResult Get(string? name = "N/A")
{
var log = new ActivityLog()
{
CreationTime = DateTime.UtcNow,
Action = "Home page visit",
IPAddress = HttpContext.Connection.RemoteIpAddress ?? IPAddress.Any,
Metadata = $"{{ name: {name} }}",
UserId = User.FindFirstValue(ClaimTypes.NameIdentifier)
};
queue.Add(log);
return Ok("Welcome!");
}
As I explained in the comments, I want to get as many as items that are available and save them with one DB trip.正如我在评论中解释的那样,我想获得尽可能多的可用项目并通过一次 DB 行程保存它们。 My solution was the following code, but it doesn't work:
我的解决方案是以下代码,但它不起作用:
while (!queue.IsCompleted)
{
var items = queue.GetConsumingEnumerable();
dbContext.AddRange(items);
dbContext.SaveChanges();
}
So instead, I'm using this which does a DB trip per each row:所以相反,我使用它每行执行一次数据库旅行:
foreach (var item in queue.GetConsumingEnumerable())
{
dbContext.Add(item);
dbContext.SaveChanges();
}
I also have two side questions: How can I increase workers?我还有两个附带的问题:如何增加工人? How can I increase workers dynamically based on queue count?
如何根据队列数动态增加工作人员?
I created an example project for this question:我为这个问题创建了一个示例项目:
This is what I ended up with in case anyone needs it (thanks to Theodor Zoulias 's hint):如果有人需要,这就是我最终得到的结果(感谢Theodor Zoulias的提示):
public class DbLogWorker : IHostedService
{
private const int BufferThreshold = 100;
private readonly TimeSpan BufferTimeLimit = TimeSpan.FromSeconds(20);
private DateTime lastDbUpdate = DateTime.UtcNow;
private readonly LogQueue queue;
private readonly IServiceScopeFactory scf;
private Task jobTask;
public DbLogWorker(LogQueue queue, IServiceScopeFactory scf)
{
this.queue = queue;
this.scf = scf;
jobTask = new Task(Job, TaskCreationOptions.LongRunning);
}
private void Job()
{
using var scope = scf.CreateScope();
var dbContext = scope.ServiceProvider.GetRequiredService<ApplicationDbContext>();
ActivityLog nextItem;
try
{
while (queue.TryTake(out nextItem, -1))
{
var buffer = new List<ActivityLog>();
buffer.Add(nextItem);
while (buffer.Count < BufferThreshold &&
DateTime.UtcNow - lastDbUpdate < BufferTimeLimit)
{
if (queue.TryTake(out nextItem))
buffer.Add(nextItem);
}
dbContext.AddRange(buffer);
dbContext.SaveChanges();
lastDbUpdate = DateTime.UtcNow;
}
}
catch (ObjectDisposedException)
{
// queue completed?
}
}
public Task StartAsync(CancellationToken cancellationToken)
{
jobTask.Start(TaskScheduler.Default);
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
queue.Complete();
jobTask.Wait(); // or 'await job' ?
return Task.CompletedTask; // unnecessary if I use 'await job'
}
}
I can recommend you to write your worker as follows, looks like it will help you:我可以推荐你写你的工人如下,看起来它会帮助你:
public class ActivityWorker : IWorker
{
// logs
private static readonly Logger Logger = LogManager.GetLogger(nameof(ActivityWorker));
// services (if necessary)
private readonly BlockingCollection<ActivityLog> _tasks = new BlockingCollection<ActivityLog>();
private Task _processingTask;
public ActivityWorker(// your services if required)
{
}
public void Start()
{
_processingTask = Task.Factory.StartNew(Process, TaskCreationOptions.LongRunning);
}
public void Stop()
{
_tasks.CompleteAdding();
_processingTask?.Wait();
}
// push your activity log here to collection
public void Push(ActivityLog task)
{
_tasks.Add(task);
}
private void Process()
{
try
{
while (!_tasks.IsCompleted)
{
var task = _tasks.Take();
ProcessTask(task);
}
}
catch (InvalidOperationException)
{
Logger.Warn($"ActivityLog tasks worker have been stopped");
}
catch (Exception ex)
{
Logger.Error(ex);
}
}
private void ProcessTask(ActivityLog task)
{
// YOUR logic here
}
public void Dispose()
{
_tasks?.CompleteAdding();
_processingTask?.Wait();
_processingTask?.TryDispose();
_tasks?.TryDispose();
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.