Using parallelism to speed up the code

Question

I've made a tool that sends HTTP requests (GET) (reads the info of what to send from a .txt), captures the json and parses it + writes it to a .txt. The tool is a console app targetting .NET Framework 4.5.

Could I possibly speed this up with the help of "multi-threading"?

        System.IO.StreamReader file =
            new System.IO.StreamReader(@"file.txt");
        while ((line = file.ReadLine()) != null)
        {
            using (var client = new HttpClient(new HttpClientHandler { AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate }))
            {
                client.BaseAddress = new Uri("https://www.website.com/");
                HttpResponseMessage response = client.GetAsync("index?userGetLevel=" + line).Result;
                response.EnsureSuccessStatusCode();
                string result = response.Content.ReadAsStringAsync().Result;
                dynamic json = JObject.Parse(result);
                string level = json.data.level;

                if (level == "null")
                {
                    Console.WriteLine("{0}: User does not exist.", line);
                }
                else
                {
                    Console.WriteLine("{0}: User level is {1}.", line, level);

                    using (StreamWriter sw = File.AppendText(levels.txt))
                    {
                        sw.WriteLine("{0} : {1}", line, level);
                    }
                }

            }
        }

        file.Close();

---

Answers to questions:

"Speed up what?": I'd like to speed up the whole process (the amount of requests it sends each time, not how fast it sends them.

"How many requests?": The tool reads a string from a text file and puts that information in to a part of the URL, and then captures the response, then places that in a result.txt. I'd like to increase the speed it does this at/how many it does at a time.

"Can the requests happen concurrently or are dependant on prior request responses?": Yes, and no, they are not dependent.

" Does your web server impose a limit on the number of concurrent requests?": No.

"How long does a typical request take?": The request + the time the response shows up on console per each requests is a little bit more than 1/3 of a second.

Answer 1

There are several ways to asymmetric programming, one of them could be something like this:

var messages = new ConcurrentQueue<string>();
//or
//var lockObj = new object();

public int main()
{
    var fileText = File.ReadAllLines(@"file.txt");

    var taskList = new List<Task>();
    foreach (var line in fileText)
    {
        taskList.Add(Task.Factory.StartNew(HandlerMethod, line));
        //you can control the amount of produced task if you want:
        //if(taskList.Count > 20)
        //{
        //    Task.WaitAll(taskList.ToArray());
        //    taskList.Clear();
        //}
    }
    Task.WaitAll(taskList.ToArray()); //this line may not work as I expected.

    //for the first way
    var results = new StringBuilder();
    foreach (var msg in messages)
    {
        results.AppendLine("{0} : {1}", line, level);
    }
    File.WriteAllText("path", results.ToString());
}

For writing the results, either you can use a public concurrent collection or use a lock pattern :

public void HandlerMethod(object obj)
{
    var line = (string)obj;
    var result = string.Empty;
    using (var client = new HttpClient(new HttpClientHandler { AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate }))
    {
        client.BaseAddress = new Uri("https://www.website.com/");
        HttpResponseMessage response = client.GetAsync("index?userGetLevel=" + line).Result;
        response.EnsureSuccessStatusCode();
        string result = response.Content.ReadAsStringAsync().Result;
        dynamic json = JObject.Parse(result);
        result = json.data.level;
    }

    //for the first way
    if (string.IsNullOrWhiteSpace(result))
    {
        messages.Enqueue("{0}: User does not exist.", line);
    }
    else
    {
        messages.Enqueue("{0}: User level is {1}.", line, result);
    }

    //for the second way
    //lock(lockObj)
    //{
    //    using (StreamWriter sw = File.AppendText(levels.txt))
    //    {
    //        sw.WriteLine("{0} : {1}", line, level);
    //    }
    //}
}