Why does my IIS application pool shutdown after throws and exception by worker thread while using multi threads?

Question

I am trying to create some email sending process. Without the multi-threading approach, it is working as expected. But it took more time to send all emails.

After I decided to use multi-threading and I also do some exception handling for the multi-threading process as well.

Here are my methods,

public enum EmailRequestProcesStatus
{
    Pending = 1,
    Processing = 2,
    Complete = 3
}
public async Task<int> InvokeSendingEmailAsync(string serverFolderPath)

{
            // update pending requests
            await UpdateEmailRequestToProcessAsync(EmailRequestProcesStatus.Processing);

            var invokedDateTime = DateTime.Now;
            // first get the CustomerEmail list
            var emailList= await GetPendingEmailList();

            IList<CustomerRule> customerRules = new List<CustomerRule>();
            int ruleLength = 0;
            List<Thread> threadList = new List<Thread>();
            // email count list
            List<int> sendEmailCountList = new List<int>();

            foreach (var customer in emailList)
            {
                ruleLength++;
                customerRules.Add(customer);
                if (ruleLength % 20 == 0)
                {
                    var custList = customerRules;
                    Thread t = new Thread(() => ProcessEmailSend(custList, serverFolderPath, invokedDateTime, sendEmailCountList));
                    threadList.Add(t);
                    t.Start();
                    customerRules = new List<CustomerRule>();
                }
            }

            if (customerRules.Count > 0)
            {
                Thread t = new Thread(() => ProcessEmailSend(customerRules, serverFolderPath, invokedDateTime, sendEmailCountList));
                threadList.Add(t);
                t.Start();
            }

            // join the all threads
            foreach (var thread in threadList)
            {
                thread.Join();
            }

            return sendEmailCountList.Count;
}

private void ProcessEmailSend(IList<CustomerRule> customerList, string serverFolderPath, DateTime invokedDateTime, List<int> sendEmailCountList)
{
   foreach(var cust in customerList)
   {
       var customerDetailList = get customerEmailDetails(cust);

       // access some file from file server by giving details
       var result = getPdf(customerDetailList.files);

       // do the rest of work using 
   }
}

private filesDto getPdf(filesDto files)
{
   try
   {
      //here I call the file server 
   }
   catch(Exception e)
   {
       UpdateEmailServiceWhenException();
       throw;
   }
}

After the throw, it will be captured by the global exception handler. If one worker thread got an exception for file not found from the file server then it inserted to the table as an exception and throw again.

From that, all the processes will be terminated and the IIS app pool also shutdown. There can be ambiguous multiple records are in the table as well.

How do I sort it?

Answer 1

Most likely, multiple unhandled exceptions being thrown in that worker thread results in the Rapid-Fail Protection feature.
https://docs.microsoft.com/en-us/iis/configuration/system.applicationHost/applicationPools/add/failure
Simply speaking, enough unhandled exceptions in a configurable timespan (the default is 5 in 5 minutes) will shut down the App Pool, causing the 503 Service Unavailable response.
Please check the advanced setting in the Application pool.

To fix this, we need to handle the exception or prevent the exception from being thrown.
Please refer to the below discussion.
Why does my IIS7 application pool shutdown after an exception in a DLL called from an ASP.NET page?