Retrieve information from a REST service asynchronously while throttling the number of requests

Question

I have a program that needs to get some data from an Atom feed. I tried two approaches and neither of them worked well.

I've used WebClient to synchronously download all the posts I need, but as there are a few thousand and the service is slow it takes many hours.

I've tried (for the first time) async/await, the new HttpClient and Task.WhenAll. Unfortunately that results in thousands of requests hitting the service and bringing it down.

How can I run say 100 requests in parallel?

Answer 1

You could use Parellel with ParallelOptions.MaxDegreeOfParallelism

ParallelOptions.MaxDegreeOfParallelism Property

Or a BlockingCollection with a bounded collection size

BlockingCollection Overview

I would recommend the BlockingCollection

Answer 2

Sounds like you have a solution already in that you can get a lot done at once. I'd suggest just adding another layer on top of that which just loops through all of the posts, but only processes them 100 at a time.

Right now you might have: DownloadAll(List ListofPosts) Inside of DownloadAll you probably have a wait all at the end.

instead: For loop from 1 to ( ListofPosts Count / 100) DownloadAll(ListofPosts.Skip(xxx).Take(100));

Obviously not real code there, but then you can do chunks of 100 with little change to your main function.