How to use multi threading in a For loop

Question

I want to achieve the below requirement; please suggest some solution.

string[] filenames = Directory.GetFiles("C:\Temp"); //10 files

for (int i = 0; i < filenames.count; i++)    
{
    ProcessFile(filenames[i]); //it takes time to execute    
}

I wanted to implement multi-threading. eg There are 10 files. I wanted to process 3 files at a time (configurable, say maxthreadcount ). So 3 files will be processed in 3 threads from the for loop and if any thread completes the execution, it should pick the next item from the for loop. Also wanted to ensure all the files are processed before it exits the for loop.

Please suggest best approach.

Answer 1

Try

Parallel.For(0, filenames.Length, i => {
    ProcessFile(filenames[i]);
});

MSDN

It's only available since .Net 4. Hope that acceptable.

Answer 2

This will do the job in .net 2.0:

class Program
{

    static int workingCounter = 0;
    static int workingLimit = 10;
    static int processedCounter = 0;

    static void Main(string[] args)
    {
        string[] files = Directory.GetFiles("C:\\Temp");
        int checkCount = files.Length;
        foreach (string file in files)
        {
            //wait for free limit...
            while (workingCounter >= workingLimit)
            {
                Thread.Sleep(100);
            }
            workingCounter += 1;
            ParameterizedThreadStart pts = new ParameterizedThreadStart(ProcessFile);
            Thread th = new Thread(pts);
            th.Start(file);
        }
        //wait for all threads to complete...
        while (processedCounter< checkCount)
        {
            Thread.Sleep(100);
        }
        Console.WriteLine("Work completed!");
    }

    static void ProcessFile(object file)
    {
        try
        {
            Console.WriteLine(DateTime.Now.ToString() + " recieved: " + file + " thread count is: " + workingCounter.ToString());
            //make some sleep for demo...
            Thread.Sleep(2000);
        }
        catch (Exception ex)
        {
            //handle your exception...
            string exMsg = ex.Message;
        }
        finally
        {
            Interlocked.Decrement(ref workingCounter);
            Interlocked.Increment(ref processedCounter);
        }
    }
}

Answer 3

Take a look at the Producer/Consumer Queue example by Joe Albahari. It should provide a good starting point for what you're trying to accomplish.

Answer 4

You could use the ThreadPool .

Example:

ThreadPool.SetMaxThreads(3, 3);

for (int i = 0; i < filenames.count; i++)    
{
    ThreadPool.QueueUserWorkItem(new WaitCallback(ProcessFile), filenames[i]);
}

static void ProcessFile(object fileNameObj)
{
    var fileName = (string)fileNameObj;
    // do your processing here.
}

---

If you are using the ThreadPool elsewhere in your application then this would not be a good solution since it is shared across your app.

You could also grab a different thread pool implementation, for example SmartThreadPool

Answer 5

Rather than starting a thread for each file name, put the file names into a queue and then start up three threads to process them. Or, since the main thread is now free, start up two threads and let the main thread work on it, too:

Queue<string> MyQueue;

void MyProc()
{
    string[] filenames = Directory.GetFiles(...);
    MyQueue = new Queue(filenames);

    // start two threads
    Thread t1 = new Thread((ThreadStart)ProcessQueue);
    Thread t2 = new Thread((ThreadStart)ProcessQueue);
    t1.Start();
    t2.Start();

    // main thread processes the queue, too!
    ProcessQueue();

    // wait for threads to complete
    t1.Join();
    t2.Join();
}

private object queueLock = new object();

void ProcessQueue()
{
    while (true)
    {
        string s;
        lock (queueLock)
        {
            if (MyQueue.Count == 0)
            {
                // queue is empty
                return;
            }
            s = MyQueue.Dequeue();
        }
        ProcessFile(s);
    }
}

Another option is to use a semaphore to control how many threads are working:

Semaphore MySem = new Semaphore(3, 3);

void MyProc()
{
    string[] filenames = Directory.GetFiles(...);

    foreach (string s in filenames)
    {
        mySem.WaitOne();
        ThreadPool.QueueUserWorkItem(ProcessFile, s);
    }

    // wait for all threads to finish
    int count = 0;
    while (count < 3)
    {
        mySem.WaitOne();
        ++count;
    }
}

void ProcessFile(object state)
{
    string fname = (string)state;
    // do whatever
    mySem.Release();  // release so another thread can start
}

The first will perform somewhat better because you don't have the overhead of starting and stopping a thread for each file name processed. The second is much shorter and cleaner, though, and takes full advantage of the thread pool. Likely you won't notice the performance difference.

Answer 6

Can set max threads unsing ParallelOptions

Parallel.For Method (Int32, Int32, ParallelOptions, Action)

ParallelOptions.MaxDegreeOfParallelism

Answer 7

var results = filenames.ToArray().AsParallel().Select(filename=>ProcessFile(filename)).ToArray();

bool ProcessFile(object fileNameObj)
{
    var fileName = (string)fileNameObj;

    // do your processing here.

    return true;
}