BackgroundWorker或Task.WhenAll

Question

I am using a background service now to process a time consuming operation and same time showing progress in UI

But now looking for some performance improvements in terms of time [since at present background worker took more time to complete the processings]

The current solution of Do_Work in background worker is like

foreach(string _entry in ArrayList){

  //process the _entry(communicate to a service and store response in Db) and took almost 2-3 seconds for the completion
}

ArrayList contains almost 25000 records . So almost 25000*3 = 75000 seconds time is required to do the processing now

The new solution i am thinking of is like starting 50 Threads of 500 items each at same time and waits for the completion of all Threads something like this

int failed = 0;
var tasks = new List<Task>();            
for (int i = 1; i < 50; i++)
{
    tasks.Add(Task.Run(() =>
    {

        try
        {
            //Process 500 items from Array .(communicate to a service and store response in Db)
        }
        catch
        {
            Interlocked.Increment(ref failed);
            throw;
        }
    }));
}
Task t = Task.WhenAll(tasks);    //Runs 50 Threads         
try
{
    await t;
}
catch (AggregateException exc)
{
string _error="";
    foreach (Exception ed in t.Exception.InnerExceptions)
    {
       _error+=ed.Message;
    }
     MessageBox.Show(_error);
}
if (t.Status == TaskStatus.RanToCompletion)
    MessageBox.Show("All ping attempts succeeded.");
else if (t.Status == TaskStatus.Faulted)
    MessageBox.Show("{0} ping attempts failed", failed.ToString());

Will this helps to reduce the processing time? Or some better approaches? I tried with a small sample size of 10 and debugged , but i cant see much difference (Something is wrong in my choice of WhenAll ?)

Answer 1

You need to handle few records at a time.

Process n records with n different tasks. Let's say n = 5. So 5 different tasks t1,t2,t3,t4 and t5. Once anyone is completed processing data one row they should start processing t (1+n), t (2+n), t(3+n),t(4+n), t(5+n) row and so on until all records are processed.

Store all those processed values into a dictionary or list to identify which one belongs to which record.

It's like recursive function.

I have used this kind of approach in past and it really improves the performance a lot. You can fine-tune the value of n based on your PC configuration

Answer 2

我建议尝试使用BlockingCollection