Parallel.ForEach and blocking thread

Question

I created Windows Service application with Quartz.NET library to schedule jobs for reporting purposes. Main part of application is fetching some data from databases on different locations (~260), so I decided to use Parallel.ForEach for parallel fetching and storing data on central location.

In Quartz.NET Job I run static method from my utility class that do parallel processing.

Utility class:

public class Helper
{
    public static ConcurrentQueue<Exception> KolekcijaGresaka = new ConcurrentQueue<Exception>();   // Thread-safe

    public static void Start()
    {
        List<KeyValuePair<string, string>> podaci = Aktivne();  // List of data for processing (260 items)
        ParallelOptions opcije = new ParallelOptions { MaxDegreeOfParallelism = 50 };
        Parallel.ForEach(podaci, opcije, p =>
        {
            UzmiPodatke(p.Key, p.Value, 2000);
        });
    }

    public static void UzmiPodatke(string oznaka, string ipAdresa, int pingTimeout)
    {
        string datumTrenutneString = DateTime.Now.ToString("d.M.yyyy");
        string datumPrethodneString = DatumPrethodneGodineString();
        string sati = DateTime.Now.ToString("HH");

        // Ping:
        Ping ping = new Ping();
        PingReply reply = ping.Send(ipAdresa, pingTimeout);

        // If is online call method for copy data:
        if (reply.Status == IPStatus.Success)
        {
            KopirajPodatke(oznaka, ipAdresa, datumTrenutneString, datumPrethodneString, sati, "TBL_DATA");
        }
    }
    public static void KopirajPodatke(string oznaka, string ipAdresa, string datumTrenutneString, string datumPrethodneString, string sati, string tabelaDestinacija)
    {
        string lanString = "Database=" + ipAdresa + "://DBS//custdb.gdb; User=*******; Password=*******; Dialect=3;";
        IDbConnection lanKonekcija = new FbConnection(lanString);
        IDbCommand lanCmd = lanKonekcija.CreateCommand();

        try
        {
            lanKonekcija.Open();
            lanCmd.CommandText = "query ...";
            DataTable podaciTabela = new DataTable();

            // Get data from remote location:
            try
            {
                podaciTabela.Load(lanCmd.ExecuteReader());
            }
            catch (Exception ex)
            {
                throw ex;
            }

            // Save data:
            if (podaciTabela.Rows.Count > 0)
            {
                using (SqlConnection sqlKonekcija = new SqlConnection(Konekcije.DB("Podaci")))
                {
                    sqlKonekcija.Open();
                    using (SqlBulkCopy bulkcopy = new SqlBulkCopy(sqlKonekcija))
                    {
                        bulkcopy.DestinationTableName = tabelaDestinacija;
                        bulkcopy.BulkCopyTimeout = 5;  // seconds
                        bulkcopy.ColumnMappings.Add("A", "A");
                        bulkcopy.ColumnMappings.Add("B", "B");
                        bulkcopy.ColumnMappings.Add("C", "C");
                        bulkcopy.ColumnMappings.Add("D", "D");
                        try
                        {
                            bulkcopy.WriteToServer(podaciTabela);
                        }
                        catch (Exception ex)
                        {
                            throw ex;
                        }
                    }
                }
            }
        }
        catch (Exception ex)
        {
            KolekcijaGresaka.Enqueue(ex);
        }
        finally
        {
            lanCmd.Dispose();
            lanKonekcija.Close();
            lanKonekcija.Dispose();
        }
    }

Application works most of times (job is executing 4 times per day), but sometimes get stuck and hanging (usually when processed ~200 items parallel) thus blocking main thread and never ends. Seems like one of thread from parallel processing get blocked and prevents execution of main thread. Can this be caused by deadlocks?

How can I ensure that no one thread blocks application execution (even with no success of fetching data)? What can get wrong with code above?

Answer 1

How can I ensure that no one thread blocks application execution (even with no success of fetching data)? What can get wrong with code above?

Parallel.Foreach is not asynchronous, it only executes each iteration in parallel, so it will wait for every operation to finish before proceeding. If you truly do not care to wait for all operations to finish before proceeding back to the caller, then try using the Task factory to schedule these and use the thread pool by default.

ie

foreach(var p in podaci)
{
    Task.Factory.StartNew(() => UzmiPodatke(p.Key, p.Value, 2000));
}

Or use ThreadPool.QueueUserWorkItem or BackgroundWorker, whatever you're familiar with and is applicable to the behavior you want.

This probably won't solve all your problems , just the unresponsive program. Most likely, if there is actually a problem with your code, one of your Tasks will eventually throw an exception which will crash your program if unhandled. Or worse yet, you will have "stuck" tasks just sitting there hogging resources if the Task(s) never finish. However, it may just be the case that occasionally one of these takes extremely long. In this case, you can handle this however you want (cancellation of long task, make sure all previously scheduled tasks complete before scheduling more, etc.), and the Task Parallel Library can support all these cases with some minor modifications.