Bulk Export Data From Three Tables Into One Excel Sheet

Question

The company that I am working for requires their inventory to be exported to an excel sheet on a regular basis. Thankfully it is less than 50k items in total, but with how I am currently exporting, it is taking an absurd amount of time. The items are split into three different tables, but I think my issue comes from having to open multiple connections per item to get information such as converting brandID to brandName, or locationID to locationName.

Note that all code is writen in c#. Here is the current method that I am using to start the exporting:

  public void exportAllItems()
    {
        exportTable = new System.Data.DataTable();

        exportTable.Columns.Add("Vendor", typeof(string));
        exportTable.Columns.Add("Store_ID", typeof(string));
        exportTable.Columns.Add("ItemNumber", typeof(string));
        exportTable.Columns.Add("Shipment_Date", typeof(string));
        exportTable.Columns.Add("Brand", typeof(string));
        exportTable.Columns.Add("Model", typeof(string));
        exportTable.Columns.Add("Club_Type", typeof(string));
        exportTable.Columns.Add("Shaft", typeof(string));
        exportTable.Columns.Add("Number_of_Clubs", typeof(string));
        exportTable.Columns.Add("Tradein_Price", typeof(double));
        exportTable.Columns.Add("Premium", typeof(double));
        exportTable.Columns.Add("WE PAY", typeof(double));
        exportTable.Columns.Add("QUANTITY", typeof(int));
        exportTable.Columns.Add("Ext'd Price", typeof(double));
        exportTable.Columns.Add("RetailPrice", typeof(double));
        exportTable.Columns.Add("Comments", typeof(string));
        exportTable.Columns.Add("Image", typeof(string));
        exportTable.Columns.Add("Club_Spec", typeof(string));
        exportTable.Columns.Add("Shaft_Spec", typeof(string));
        exportTable.Columns.Add("Shaft_Flex", typeof(string));
        exportTable.Columns.Add("Dexterity", typeof(string));
        exportTable.Columns.Add("Destination", typeof(string));
        exportTable.Columns.Add("Received", typeof(string));
        exportTable.Columns.Add("Paid", typeof(string));

        exportAllAdd_Clubs();
        exportAllAdd_Accessories();
        exportAllAdd_Clothing();

        DataColumnCollection dcCollection = exportTable.Columns;

        // Export Data into EXCEL Sheet
        Application ExcelApp = new Application();
        ExcelApp.Application.Workbooks.Add(Type.Missing);          

        for (int i = 1; i < exportTable.Rows.Count + 2; i++)
        {
            for (int j = 1; j < exportTable.Columns.Count + 1; j++)
            {
                if (i == 1)
                {
                    ExcelApp.Cells[i, j] = dcCollection[j - 1].ToString();
                }
                else
                    ExcelApp.Cells[i, j] = exportTable.Rows[i - 2][j - 
1].ToString();
            }
        }
        //Get users profile, downloads folder path, and save to workstation
        string pathUser = 
Environment.GetFolderPath(Environment.SpecialFolder.UserProfile);
        string pathDownload = Path.Combine(pathUser, "Downloads");
        ExcelApp.ActiveWorkbook.SaveCopyAs(pathDownload + "\\TotalInventory-
" + DateTime.Now.ToString("d MMM yyyy") + ".xlsx");
        ExcelApp.ActiveWorkbook.Saved = true;
        ExcelApp.Quit();
    }

Due to the requirements of the company, the excel sheet needs to have headers, and the same information as the datatable that is created above. After the creation of the exportTable , I call three other methods that go through each item table. I'll show just one of them because they are mostly the same:

//Puts the clubs in the export table
    public void exportAllAdd_Clubs()
    {
        SqlConnection conn = new SqlConnection(connectionString);
        SqlCommand cmd = new SqlCommand();
        cmd.Connection = conn;
        cmd.CommandText = "Select * from tbl_clubs";
        conn.Open();
        SqlDataReader reader = cmd.ExecuteReader();
        while (reader.Read())
        {             
            exportTable.Rows.Add("", 
(lm.locationName(Convert.ToInt32(reader["locationID"]))).ToString(), 
(Convert.ToInt32(reader["sku"])).ToString(),
                "", idu.brandType(Convert.ToInt32(reader["brandID"])), 
idu.modelType(Convert.ToInt32(reader["brandID"])), 
reader["clubType"].ToString(),
                reader["shaft"].ToString(), 
reader["numberOfClubs"].ToString(), 0, Convert.ToDouble(reader["premium"]), 
Convert.ToDouble(reader["cost"]),
                Convert.ToInt32(reader["quantity"]), 0, 
Convert.ToDouble(reader["price"]), reader["comments"].ToString(), "", 
reader["clubSpec"].ToString(),
                reader["shaftSpec"].ToString(), 
reader["shaftFlex"].ToString(), reader["dexterity"].ToString(), "", "", "");
        }
        conn.Close();
    }

Inside of the above method, I have to make three calls to different methods to convert the data to a meaningful string. One for the brand, model, and location.

I am open to all sorts of ideas on how to speed this up or properly go about bulk exporting.

Thank you for your time!

Update

After messing around with EPPlus, my export times were still taking way too long. I think it has to do with how I am going through each row in the database and exporting them one at a time. Still trying to find a way to do it differently.

Answer 1

I figured out what the problem was:

Instead of having one giant query with sub queries, I was doing multiple queries per item which was taking way too much time. After remembering that I could do sub queries, and implementing the changes, the export time dropped from over 30 minutes to just under 5 seconds.