webrequest c# million records

Question

I need some suggestions for some things that I am doing that's taking a 'long time' to run.

1. I have about 100000 records in the database which has a date column. This needs to be updated.
2. This is done by calling a URL for each record and getting date and updating it.
3. Right now, I have written C# code that selects 1000 records at a time.

Code:

WebRequest wr = WebRequest.Create(inputURL);
HttpWebResponse response = (HttpWebResponse)wr.GetResponse();

// Get the stream containing content returned by the server.
Stream dataStream = response.GetResponseStream();

// Read the content. 
StreamReader reader = new StreamReader(dataStream);

string responseString = reader.ReadToEnd();

The response is in the form of json and so using Jsonserializer I get the date. And then I update one record at a time.

Now, this is taking a very long time. How can I improve performance of this??

Answer 1

Can you execute sql using SSMS or another method? You haven't specified the rules for updating the date, but if you had a truly simply case, say adding 10 days to every date, you could execute that with one line of code in milliseconds, ie

update mytable set mydate = dateadd(day, 10, mydate) ...voila.

even if you have more complicated rules for updating, you are likely better of doing this thru sql than thru a webrequest one record at a time...

What are the rules for updating the date?

Answer 2

You can open many URLs to the server simultaneously. An easy way would be to call this method using TPL (Task Parallelism Library) in .NET (V4 and higher I think, maybe 3.5)

List<string> Urls = GetMyList(); 
System.Threading.Tasks.Parallel.ForEach( Urls, u=>Doit(u) );

For example, look at the answer here: Parallel Extensions

You can limit the number of threads which would be active at the same time.