Tree Hierarchy of WebClient Requests using TPL

Question

I am developing a WPF app in C# where I have an Uri that I want to download Json data. The code will deserialize the downloaded Json data to object, thereafter, the object has a list of Uris that would require to request more Json data which i would like to request in parallel. And the downloaded Json data might have more Uris to request. The code should be able to do WebClient.CancelAsync on the parent and its children's WebClient when desired. I am looking at the Task Parallel Library and finding it difficult to grasp and implement it. I'm unsure if I should use the TPL's Cancellation token to invoke the CancelAsync or the CancelAsync to cancel the TPL's Cancellation token. And I not sure if i should use nested Tasks for the children WebClient....?

If anyone has a similar scenario and implemented it using the new TPL.. would you mind sharing a snippet of the code...

Thanks.

Answer 1

If I can suggest using Reactive Extensions (Rx) on top of the TPL then this can be done quite easily without the need for cancellation tasks etc.

If I can assume you have the following:

// The initial Uri
Uri initialUri = ...;

// A function to return the JSON string from a given Uri
Func<Uri, string> getJason = ...; 

// Turn the JSON into the next set of Uris to fetch
Func<string, IEnumerable<Uri>> getUris = ...; 

Then, using Rx, you turn these functions into functions that return observables using the Task Pool, like so:

Func<Uri, IObservable<string>> getJasonObsFunc = u =>
    Observable
        .FromAsyncPattern<Uri, string>(
            getJason.BeginInvoke,
            getJason.EndInvoke)
        .Invoke(u)
        .ObserveOn(Scheduler.TaskPool);

Func<string, IObservable<Uri>> getUrisObsFunc = j =>
    Observable
        .FromAsyncPattern<string, IEnumerable<Uri>>(
            getUris.BeginInvoke,
            getUris.EndInvoke)
        .Invoke(j)
        .ObserveOn(Scheduler.TaskPool)
        .SelectMany(xs => xs.ToObservable());

You'll need a callback to get the Uri/JSON pairs out. Something like this:

Action<Uri, string> callback = (u, j) =>
    Console.WriteLine(String.Format("{0} => {1}", u, j));

Here's the recursive LINQ query that will recursively fetch each JSON string:

Func<Uri, IObservable<Uri>> getAllUris = null;
getAllUris = u =>
    Observable
        .Return<Uri>(u)
        .Merge(
            from j in getJasonObsFunc(u).Do(k => callback(u, k))
            from u1 in getUrisObsFunc(j)
            from u2 in getAllUris(u1)
            select u2);

Then you invoke all of this goodness using the following line:

var subscription = getAllUris(initialUri).Subscribe();

Now, if you want to cancel the query execution just call this:

subscription.Dispose();

Rx handles all of the tasks and cancels them all for you.

I hope this helps.

Here are the links for Rx:

• Reactive Extensions Forum
• Reactive Extensions (Rx) v1.0.10621

Answer 2

我建议您避免创建嵌套任务，因为您无法控制正在运行的任务的数量，而是可以使用BlockingCollection使用分派模式，因为其中的工作任务数量有限，它们可以从集合中分派工作，并且可以添加如有必要，还需要做更多的工作，并且在下载所有对象时，请调用CompleteAdding以取消阻止所有等待的任务。