C＃并行处理多个任务

Question

I want to achieve something like :

• Create a bunch of mongo queries

• For each query, get the deserialized mongo data set as a ObjectA

• Map all ObjectA to a list of ObjectB (takes some time given that we're working with several millions objects)

• When the mapping is done, upserts all ObjectB (in a loop) to a new mongo db/collection (also takes some time), and start fetching/mapping the next data set with the next query

• if the next mapping is done before the last mongo upsert is finished, wait for it to complete then start writing that data-set to mongo

Now I've been toying around with Task , I think it's the way to go but I have a hard time getting how to do this.

Should I run 2 arrays of tasks?

How can I create the "dependency" between a "fetch/map task" and a "write to mongo task"?

Any help appreciated, thanks!

Answer 1

Have a look at Dataflow http://msdn.microsoft.com/en-us/library/hh228603(v=vs.110).aspx

You can create a block for each stage of your process and then chain them together.

1. execute query (TransformBlock)
2. transform response (TransformBlock)
3. upsert (ActionBlock)

then you can post the unique query data to the dataflow in a loop and set the parallelism as needed. eg up the parallelism of stage 1 & 2 as long a you aren't having memory issues, but keep stage 3 non-parallel