TPL Parallel.ForEach中的每个线程实例对象

Question

Is there a TPL syntax which allows you to inject objects from pool into tasks so that one object is only used by one thread at once? Or even better - only used by the same once thread?

Usage example

Assume I want to create 10 threads which open 10 files: 1.txt , 2.txt , 3.txt ... 10.txt and write 500 000 consequent numbers randomly to these files.

I can do this:

ConcurrentQueue<int> objs = new ConcurrentQueue<int>(); // 500000 numbers go here
Task[] tasks = Enumerable.Range(1, 10)
    .Select(i =>
    {
        return Task.Factory.StartNew(() => 
        {
            using (var f = File.Open($"{i}.txt"))
            {
                using (var wr = StreamWriter(f))
                {
                    while (objs.TryDequeue(out int obj))
                    {
                        wr.WriteLine(obj);
                    }
                }
            }
        }
    })
    .ToArray();
Task.WaitAll(tasks);

However, is it possible to provide the same behaviour without utilizing concurrent collections, just with TPL?

Answer 1

It would be better if everything except the last two edits was removed.

If the question is Can you pass an object per task (not thread) when using Parallel. ? The answer is : Yes you can, through any of the overloads that accept local state, ie have a TLocal type like this one :

public static ParallelLoopResult ForEach<TSource, TLocal>(
    IEnumerable<TSource> source,
    Func<TLocal> localInit,
    Func<TSource, ParallelLoopState, TLocal, TLocal> body,
    Action<TLocal> localFinally
)

Parallel.For doesn't use threads. It partitions the data and creates one task for each partitions. Each task ends up processing all of a partition's data. Typically, Parallel uses as many tasks as there are cores. It also uses the current thread for processing, which is why it appears to block the current thread. It doesn't, it's begin used to process one of the partitions.

The functions that deal with local data allow you to generate an initial local value and pass it to each body invocation. All overloads with local data require the body to retun the (possibly modified) data, so Parallel itself doesn't have to store it. This is essential, since Parallel. can terminate and restart tasks. It wouldn't be able to do so easily or efficiently if it had to keep track of local data.

For this particular example, and bypassing the fact that ORMs are unsuitable for bulk operations, especially when dealing with hundreds of thousands of objects, localInit should create a new session. body should use and return that session while finally, localFinally should dispose it.

var mySessionFactory
var myData=....;
Parallel.ForEach(
    myData,
    ()=>CreateSession(),
    (record,state,session)=>{
        //process the data etc.
        return session;
    },
    (session)=>session.Dispose()
);

Some more warnings though. NH keeps changes in memory until they are flushed and the cache cleared out. This will create memory issues. One solution would be to keep count and flush the data periodically. Instead of a session, the state could be a (int counter,Session session) tupple:

Parallel.ForEach(
    myData,
    ()=>(counter:0,session:CreateSession()),
    (record,state,localData)=>{
        var (counter,session)=localData;
        //process the data etc.
        if (counter % 1000 ==0)
        {
            session.Flush();
            session.Clear();
        }
        return (++counter,session);
    },
    data=>data.session.Dispose()
);

A better solution would be to batch the objects in advance, so that instead of an IEnumerable<MyRecord> the loop would work on IEnumerable<MyRecord[]> arrays. In conjuction with batched statements this would reduce the performance penalty imposed by ORMs on bulk operations.

Writing a Batch method isn't hard, but MoreLinq already provides one, available as source or a NuGet package :

var myBatches=myData.Batch(1000);
Parallel.ForEach(
    myBatches,
    ()=>CreateSession(),
    (records,state,session)=>{

        foreach(var record in records)
        {
            //process the data etc.
            session.Save(record);                
        }
        session.Flush();
        session.Clear();
        return session;
    },
    data=>data.session.Dispose()
);

Answer 2

No, there is not.

The closest solution is to create N threads manually (either with Task or Parallel.For / Parallel.ForEach ) and use ConcurrentQueue to distribute data thread-safely.