Process items in list<Object> by a set of threads - threadpooling

Question

public void multithread()
{

  List<Bbject> objectlist= new List<Object>{
      //consider it has around 10 objects
   };
  foreach (var obj in objectlist)
  {               
      Process(obj);
   }
}

private void process(Object obj)
{
    //Implementation
}

I am trying to achieve multithreading where each thread takes a object and process it... i tried creating 5 seperate threads where each thread process 2 objects...but it was more like hardconding...is there any way to achieve it using threadpool...

Answer 1

I would avoid to create Thread objects manually. Modern .NET offers more powerful tools to handle concurrency. As a rule of thumb each time you write code like new Thread in modern .NET you are almost certainly not doing the right thing.

The solution which seems to suit the best for your problem is the static method ForEach defined on the Parallel class.

Parallel.ForEach will basically run a parallel foreach loop on a provided sequence of objects. The threads used to process the items in parallel are borrowed from the thread pool which is managed directly by the .NET framework, so that you don't need to create Thread objects by yourself.

There are some caveats that you need to consider before proceed:

• the items in the provided sequence must be independent so that processing them in parallel is a safe operation
• using Parallel.ForEach makes sense when the workload done to process each item of the provided sequence is CPU-bound. For I/O workload the approach must be different (in that case you can use Task.WhenAll for instance)
• data parallelism does not come for free, there is always a computational cost in doing parallel processing. So you need to consider the trade-off and always measure the performance of your code . Generally speaking data parallelism is beneficial when the work done to process each item is not trivial and you run your code on a multi-core processor. It is totally possible that a classic sequential foreach loop leads to better performance in some cases.

That said, here is a code sample:

public static class Program 
{
  public static void Main(string[] args) 
  {
    List<object> items = ... // code omitted for brevity

    ProcessItemsInParallel(items);
  }

  private static void ProcessItemsInParallel(IEnumerable<object> items) 
  {
    Parallel.ForEach(items, ProcessItem);
  }
   
  private static void ProcessItem(object item) 
  {
    // some non - trivial CPU bound work is done here...
  }
}

This approach is known as data parallelism. Consider reading the documentation for Parallel.ForEach .

Here are some great readings on the subject of concurrency in .NET:

• Threading in C# by Joseph Albahari which is a free ebook. This book is a bit older, but it is worth reading it in order to gain the basics. It is still a great reading.
• the Stephen Cleary book about concurrency in C#. In my opinion this is the book that you need to read if you are a .NET developer and you want to learn modern patterns to handle concurrency.
• the Stephen Cleary blog is another great resource, full of useful and interesting articles.

A final consideration.

I noticed that you have used the ASP.NET core tag for your question. You did not mention where you are trying to perform your heavy CPU-bound processing.

If you are thinking to do this kind of processing in the context of handling an HTTP request (I mean inside an action method of some controller) I strongly advise against it. Doing so wil kill the scalability of your server.

Heavy CPU-bound processing should be performed in backend services; HTTP requests are meant to be served quickly.

I/O bound workload can be more easily handled in the context of serving HTTP requests by preserving server scalability. You can do so by using asynchronous action methods . This won't guarantee that the single HTTP request will be served quickly (that depends on the type of I/O work that you are going to do), but at least you will save the scalability of your server.

But please do not run a parallel foreach loop when serving HTTP requests. Use a queue to forward the workload from your web server to a backend service instead.