在我的asp.net mvc Web应用程序中使用带有WebClient（）的Parallel.Foreach有任何缺点或风险吗？

Question

I am working on an asp.net MVC-5 web application, and based on some articles i read that i should not use Parallel methods inside web servers and inside .net web applications espically. now in my case i have around 1,500 WebClient() calls that i need to issue inside a foreach, and then deserialize the return json object from the WebClient() calls. my original code before using Parallel.Foreach was as follow, which took around 15 minutes to complete:-

    public async Task <List<Details2>> Get()
            {       

              try
                {

                    using (WebClient wc = new WebClient()) 
                    {
                        string url = currentURL + "resources?AUTHTOKEN=" + pmtoken;
                        var json = await wc.DownloadStringTaskAsync(url);
                        resourcesinfo = JsonConvert.DeserializeObject<ResourcesInfo>(json);

                    }


                    ForEach( var c in resourcesinfo.operation.Details)
                   {

                        ResourceAccountListInfo resourceAccountListInfo = new ResourceAccountListInfo();
                        using (WebClient wc = new WebClient()) 
                        {

                            string url = currentURL + "resources/" + c.RESOURCEID + "/accounts?AUTHTOKEN=" + pmtoken;
                            string tempurl = url.Trim();



                            var json =  await wc.DownloadStringTaskAsync(tempurl);
                            resourceAccountListInfo = JsonConvert.DeserializeObject<ResourceAccountListInfo>(json);


                        }

                   if (resourceAccountListInfo.operation.Details.CUSTOMFIELD.Count > 0)
                    {
                        List<CUSTOMFIELD> customfield = resourceAccountListInfo.operation.Details.CUSTOMFIELD.Where(a =>
                                 a.CUSTOMFIELDLABEL.ToLower() == "name"
                                ).ToList();
                        if (customfield.Count == 1)
                        {
                            PMresourcesOnly.Add(resourceAccountListInfo.operation.Details);

                        }

                    }

                   }//end of foreach             

                    return PMresourcesOnly.ToList();

                }
                catch (Exception e)
                {
                }
                return new List<Details2>();
            }

now i did the following modifications :-

• i replace foreach with Parallel.ForEach

• since i should not use async methods inside Parallel.ForEach so i chnage the DownloadStringTaskAsync to DownloadString inside the Parallel.Foreach :-

   public async Task <List<Details2>> Get() { try { using (WebClient wc = new WebClient()) { string url = currentURL + "resources?AUTHTOKEN=" + pmtoken; var json = await wc.DownloadStringTaskAsync(url); resourcesinfo = JsonConvert.DeserializeObject<ResourcesInfo>(json); } Parallel.ForEach(resourcesinfo.operation.Details, new ParallelOptions { MaxDegreeOfParallelism = 7 }, (c) => { ResourceAccountListInfo resourceAccountListInfo = new ResourceAccountListInfo(); using (WebClient wc = new WebClient()) { string url = currentURL + "resources/" + c.RESOURCEID + "/accounts?AUTHTOKEN=" + pmtoken; string tempurl = url.Trim(); var json = wc.DownloadString(tempurl); resourceAccountListInfo = JsonConvert.DeserializeObject<ResourceAccountListInfo>(json); } if (resourceAccountListInfo.operation.Details.CUSTOMFIELD.Count > 0) { List<CUSTOMFIELD> customfield = resourceAccountListInfo.operation.Details.CUSTOMFIELD.Where(a => a.CUSTOMFIELDLABEL.ToLower() == "name" ).ToList(); if (customfield.Count == 1) { PMresourcesOnly.Add(resourceAccountListInfo.operation.Details); } } });//end of foreach return PMresourcesOnly.ToList(); } catch (Exception e) { } return new List<Details2>(); } 

now when i use the Parallel.Foreach the execution time was reduced from 15 minutes to around 7 minutes. but i am a bit confused if my second method is valid , so can anyone adivce on these questions (or any question):-

1. is using Parallel.Foreach with Webclient() a valid approach to follow ? or i should avoid using Parallel methods inside .net and web applications?

2. when using Parallel.Foreach could i face any problem such as that the return PMresourcesOnly.ToList(); is return to the client while there are still some wc.DownloadString(tempurl); that did not complete?

3. if i want to compare the 2 methods (Parallel.Foreach & Foreach) will the result be the same ?

4. on some online articles they use Task.Factory.StartNew(() instead of using Parallel.foreach so what are the main differences between them ?

EDIT I tried defining the SemaphoreSlim as follow:-

public async Task <List<Details2>> Get()
{
SemaphoreSlim throttler = new SemaphoreSlim(initialCount: 15);       
  try
  {
//code goes here

var tasks = resourcesinfo.operation.Details.Select(c => TryDownloadResourceAsync(c.RESOURCEID,throttler)).ToList();
}

///---

private async Task<Details2> TryDownloadResourceAsync(string resourceId, SemaphoreSlim throttler)
        {
            await throttler.WaitAsync();
try
            {
                using (WebClient wc = new WebClient()) //get the tag , to check if there is a server with the same name & tag..
                {}
             }
 finally
            {
                throttler.Release();
            }

Answer 1

is using Parallel.Foreach with Webclient() a valid approach to follow ? or i should avoid using Parallel methods inside .net and web applications?

No, you absolutely should avoid using parallel methods inside ASP.NET apps.

on some online articles they use Task.Factory.StartNew(() instead of using Parallel.foreach so what are the main differences between them ?

Parallel is for data parallism (running the same CPU-bound code over a collection of data items). StartNew is for dynamic task parallelism (running the same or different CPU-bound code over a collection of items that changes as you process it).

Neither approach is appropriate here, since the work you have to do is I/O-bound, not CPU-bound.

What you actually want is concurrency (doing multiple things at a time), not parallelism . Instead of using parallel concurrency (doing multiple things at a time by using multiple threads), what you want is asynchronous concurrency (doing multiple things at a time using no threads).

Asynchronous concurrency is possible in code via await Task.WhenAll , as such:

private async Task<string> TryDownloadResourceAsync(string resourceId)
{
  ResourceAccountListInfo resourceAccountListInfo = new ResourceAccountListInfo();
  using (WebClient wc = new WebClient()) 
  {
    string url = currentURL + "resources/" + resourceId + "/accounts?AUTHTOKEN=" + pmtoken;
    string tempurl = url.Trim();

    var json =  await wc.DownloadStringTaskAsync(tempurl);
    resourceAccountListInfo = JsonConvert.DeserializeObject<ResourceAccountListInfo>(json);
  }

  if (resourceAccountListInfo.operation.Details.CUSTOMFIELD.Count > 0)
  {
    List<CUSTOMFIELD> customfield = resourceAccountListInfo.operation.Details.CUSTOMFIELD.Where(a =>
        a.CUSTOMFIELDLABEL.ToLower() == "name"
    ).ToList();
    if (customfield.Count == 1)
    {
      return resourceAccountListInfo.operation.Details;
    }
  }
  return null;
}

public async Task <List<Details2>> Get()
{       
  try
  {
    using (WebClient wc = new WebClient()) 
    {
      string url = currentURL + "resources?AUTHTOKEN=" + pmtoken;
      var json = await wc.DownloadStringTaskAsync(url);
      resourcesinfo = JsonConvert.DeserializeObject<ResourcesInfo>(json);
    }

    var tasks = resourcesinfo.operation.Details.Select(c => TryDownloadResourceAsync(c.RESOURCEID)).ToList();
    var results = await Task.WhenAll(tasks).Select(x => x != null);
    return results.ToList();
  }
  catch (Exception e)
  {
  }
  return new List<Details2>(); // Please, please don't do this in production.
}

As a final note, you may want to look into HttpClient , which was designed for asynchronous operations and has the nice property that you only need one of them for any number of simultaneous calls.

Answer 2

Take a look at:
object syncObj = new object();
lock(syncObj)

try
{
    using (WebClient wc = new WebClient()) 
    {
        string url = currentURL + "resources?AUTHTOKEN=" + pmtoken;
        var json = await wc.DownloadStringTaskAsync(url);
        resourcesinfo = JsonConvert.DeserializeObject<ResourcesInfo>(json);
    }

    object syncObj = new object();  // create sync object
    Parallel.ForEach(resourcesinfo.operation.Details, new ParallelOptions { MaxDegreeOfParallelism = 7 }, (c) =>
    {
        ResourceAccountListInfo resourceAccountListInfo = new ResourceAccountListInfo();
        using (WebClient wc = new WebClient()) 
        {
            string url = currentURL + "resources/" + c.RESOURCEID + "/accounts?AUTHTOKEN=" + pmtoken;
            string tempurl = url.Trim();

            var json =  wc.DownloadString(tempurl);
            resourceAccountListInfo = JsonConvert.DeserializeObject<ResourceAccountListInfo>(json);
        }

        lock(syncObj)  // lock using sync object
        {
            PMresourcesOnly.Add(resourceAccountListInfo.operation.Details);
        }
    });//end of foreach

    return PMresourcesOnly.ToList();
}
catch (Exception e)
{
}