为什么会有这种饮食记忆？

Question

I wrote an application whose purpose is to read logs from a large table (90 million) and process them into easily understandable stats, how many, how long etc.

The first run took 7.5 hours and only had to process 27 of the 90 million. I would like to speed this up. So I am trying to run the queries in parallel. But when I run the below code, within a couple minutes I crash with an Out of Memory exception.

Environments:

Sync

Test : 26 Applications, 15 million logs, 5 million retrieved, < 20mb, takes 20 seconds

Production: 56 Applications, 90 million logs, 27 million retrieved, < 30mb, takes 7.5 hours

Async

Test : 26 Applications, 15 million logs, 5 million retrieved, < 20mb, takes 3 seconds

Production: 56 Applications, 90 million logs, 27 million retrieved, Memory Exception

public void Run()
{
    List<Application> apps;

    //Query for apps
    using (var ctx = new MyContext())
    {
        apps = ctx.Applications.Where(x => x.Type == "TypeIWant").ToList();
    }

    var tasks = new Task[apps.Count];
    for (int i = 0; i < apps.Count; i++)
    {
        var app = apps[i];
        tasks[i] = Task.Run(() => Process(app));
    }

    //try catch
    Task.WaitAll(tasks);
}

public void Process(Application app)
{
    //Query for logs for time period
    using (var ctx = new MyContext())
    {
        var logs = ctx.Logs.Where(l => l.Id == app.Id).AsNoTracking();

        foreach (var log in logs)
        {
            Interlocked.Increment(ref _totalLogsRead);

            var l = log;
            Task.Run(() => ProcessLog(l, app.Id));
        }
    }
}

Is it ill advised to create 56 contexts?

Do I need to dispose and re-create contexts after a certain number of logs retrieved?

Perhaps I'm misunderstanding how the IQueryable is working? <-- My Guess

My understanding is that it will retrieve logs as needed, I guess that means for the loop is it like a yield? or is my issue that 56 'threads' call to the database and I am storing 27 million logs in memory?

Side question

The results don't really scale together. Based on the Test environment results i would expect Production would only take a few minutes. I assume the increase is directly related to the number of records in the table.

Answer 1

With 27 Million rows the problem is one of stream processing, not parallel execution. You need to approach the problem as you would with SQL Server's SSIS or any other ETL tools: each processing step is a transofrmation that processes its input and sends its output to the next step.

Parallel processing is achieved by using a separate thread to run each step. Some steps could also use multiple threads to process multiple inputs up to a limit. Setting limits to each step's thread count and input buffer ensures you can achieve maximum throughput without flooding your machine with waiting tasks.

.NET's TPL Dataflow addresses exactly this scenario. It provides blocks to transfrom inputs to outputs (TransformBlock), split collections to individual messages (TransformManyBlock), execute actions without transformations (ActionBlock), combine data in batches (BatchBlock) etc.

You can also specify the Maximum degree of parallelism for each step so that, eg. you have only 1 log queries executing at each time, but use 10 tasks for log processing.

In your case, you could:

1. Start with a TransformManyBlock that receives an application type and returns a list of app IDs
2. A TranformBlock reads the logs for a specific ID and sends them downstream
3. An ActionBlock processes the batch.

Step #3 could be broken to many other steps. Eg if you don't need to process all app log entries together, you can use a step to process individual entries. Or you could first group them by date.

Another option is to create a custom block to read data from the database using a DbDataReader and post each entry to the next step immediatelly, instead of waiting for all rows to return. This would allow you to process each entry as it arrives, instead of waiting to receive all entries.

If each app log contains many entries, this could be a huge memory and time saver