async http requests in c#

Question

so im making a program which is kind of a web crawler. it downloads the html of a page and parses it for a specific text using regex and then adds it to a list.

to achieve this, i used async http requests. the GET request is sent asynchronously and the parsing operation is performed on the returned html.

my issue, and i'm not sure if it may be simple, is that the program doesn't run smoothly. it will send a bunch of requests, pause for a couple seconds, then increments the items parsed all at once (although the counter is programmed to increment once every time an item is added) so that for example it jumps from 53 to 69 instead of showing, 54,55,56,...

sorry for being a newb but i taught myself all this stuff and some experienced advice would go a long way.

thanks

Answer 1

That sounds correct.

The slowest part of your task is downloading the pages over the network.

Your program starts downloading a bunch of pages at once, waits for them to arrive, then parses them all almost instantly.