How to Concatenate All Worksheets of a Workbook In Excel

Question

I must first say that I am not really familiar with VBA or macros in general.

I have an Excel file that contains separate worksheets for some "Base Items".

Each worksheet contains a relationship between a product and an accessory. The number, if 0, means its incompatible, and any number > 0, is the number of accessories that can go with the product.

Each worksheet contains different accessories and codes, but some duplicates may arise.

An example of the data:

|            | J0021       | J0022        |
|:-----------|------------:|:------------:|
| Product 1  |     1       |      1       |
| Product 2  |     1       |      2       |
| Product 3  |     1       |      1       |
| Product 4  |     0       |      0       |
| Product 5  |     0       |      1       |
| Product 6  |     1       |      0       |

There are 43 separate worksheets, and anywhere from 500 to 2000 cells denoting the relationship.

I was thinking of just having a one-to-one table that denoted the relationship in the following manner:

Table Name: ProductAccessoryCompat

 Id (Primary Key)   | ProductId (Foreign Key) | AccessoryId(Foreign Key) | Count
 -------------------|-------------------------|--------------------------|------
 LB1 (ProdBase+Num) |         Product 1       |  J0021                   |   1  
 LB2                |         Product 1       |  J0022                   |   1  
 FV1                |         Product 2       |  J0021                   |   1  
 FV2                |         Product 2       |  J0022                   |   2  
 BG1                |         Product 3       |  J0021                   |   1  
 BG2                |         Product 3       |  J0022                   |   1  
 HG1                |         Product 4       |  J0021                   |   0  
 HG2                |         Product 4       |  J0022                   |   0  
 JJ1                |         Product 5       |  J0021                   |   0  
 JJ2                |         Product 5       |  J0022                   |   1  
 JL1                |         Product 6       |  J0021                   |   1  
 JL2                |         Product 6       |  J0022                   |   0  

The worksheets have the same kind of data, but will have different columns of products and different rows.

I know I need to concatenate all of them into a single worksheet, and then export it as a CSV, and then import that CSV into MySQL.

Could anyone help me determine how to concatenate the different worksheets into a single one in the format that I've described for the table I'd like to create?

I figured it would be something like the following naive pseudo-code:

var NumOfSheets = Workbook.Worksheets.Count;
var newSheet = Workbook.CreateSheet();
for (i = 0; i < NumOfSheets; i++)
{
    var curSheet = Workbook.GetSheet(i);
    curSheet.Activate;

    var numColumns = curSheet.Columns.Count;
    var numRows = curSheet.Rows.Count;
    var relationships = new Dictionary<string, string>();

    for (j = 0; j < numRows; j++)
    {
        var currentProd = curSheet.getCell(j, 0);

        for(k = 0; k < numColumns; j++)
        {
            var currentAcc = curSheet.getCell(0, k);
            var relationship = curSheet.getCell(j, k);
            if (relationship != 0) var relationships[currentProd][currentAcc] = relationship;
        }
    }
}

[Then iterate through the 2D associative array, inserting
 a new row for each product, and a new column for each
 accessory that each do net yet exist on the spreadsheet.]

I'm almost thinking it would be easier to do by hand, but you know, laziness wins.

Edit: I just realized that I could probably just ignore the 0 columns, so I added that to the pseudo-code.

Answer 1

After a bit of messing around with VBA, I decided that I did not want to go that route, and fired up Visual Studio to write my solution using C#.

A friend of mine noted that I could open and read Excel files with SQL syntax using ADO.NET.

I did this, and things worked out well, until it came time to rewrite my massaged data into a new worksheet.

There is a bug with using ADO.NET and writing into Excel that causes strings written into cells to have a an apostrophe prepended to them. (Eg "'derp" instead of "derp").

You can find information on that at this Microsoft link: How To Use ADO with Excel Data from Visual Basic or VBA

I ended up using the Office Interop classes, which I found information about here:

How to automate Microsoft Excel from Microsoft Visual C#.NET

I'm going to post my really naive attempt. Although it was successful and performed the exact function that I needed, someone might be able to look over this and pluck something useful out later.

*Note! This is just a one-off thing for me - I didn't intend on reusing this code, so it's awful. Sorry.

Without further ado:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Diagnostics;
using System.Text;
using System.Data;
using System.Data.OleDb;
using Excel = Microsoft.Office.Interop.Excel;

namespace ConsolidateAccProdExcel
{
    class Program
    {
        static void Main(string[] args)
        {
            string sheetPath = @"PATH-TO-FILE";
            string connectionString = string.Format("Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0}; Extended Properties=\"Excel 12.0 Macro;HDR=YES;IMEX=1\"",sheetPath);

            Dictionary<string, Dictionary<string, string>> relationshipDictionary = new Dictionary<string, Dictionary<string, string>>();

            using (OleDbConnection oleConnection = new OleDbConnection(connectionString))
            {
                oleConnection.Open();
                System.Data.DataTable dt = oleConnection.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);

                List<DataTable> listDt = new List<DataTable>();

                for (int i = 0; i < dt.Rows.Count; i++)
                {
                    listDt.Add(GetWorksheetData(oleConnection, (string)dt.Rows[i]["TABLE_NAME"]));
                }

                List<DataRow> rowList = new List<DataRow>();

                for (int i = 0; i < listDt.Count; i++)
                {
                    DataColumnCollection dtColumns = listDt[i].Columns;
                    foreach (DataRow drRow in listDt[i].Rows)
                    {
                        if (!(drRow.ItemArray[0] is DBNull))
                        {
                            Dictionary<string, string> accessoryData = new Dictionary<string, string>();
                            string rowName = (string)drRow.ItemArray[0];
                            for (int j = 1; j < drRow.ItemArray.Length; j++)
                            {
                                if ((string)drRow.ItemArray[j].ToString() != 0.ToString()) accessoryData.Add(dtColumns[j].ColumnName, (string)drRow.ItemArray[j].ToString());
                            }
                            try { relationshipDictionary.Add(rowName, accessoryData); }
                            catch (ArgumentException e) { Trace.WriteLine("problem: " + rowName); }
                        }
                    }
                }

                writeConcatenatedSheets(relationshipDictionary, oleConnection);
            }

            long relationshipsCount = 0;

            foreach (KeyValuePair<string, Dictionary<string, string>> product in relationshipDictionary)
            {
                relationshipsCount += product.Value.Count;
            }
            Trace.WriteLine(relationshipsCount);

        }

        private static void writeConcatenatedSheets(Dictionary<string, Dictionary<string, string>> relationshipDictionary, OleDbConnection connection)
        {
            int relationshipsCount = 0;

            foreach (KeyValuePair<string, Dictionary<string, string>> product in relationshipDictionary)
            {
                relationshipsCount += product.Value.Count;
            }

            int currentCount = 1;
            string writeCommand = "";

            Excel.Application app = new Excel.Application();
            app.Visible = true;

            Excel.Workbook workbook = app.Workbooks.Open(@"PATH-TO-FILE");
            Excel.Sheets excelSheets = workbook.Worksheets;

            Excel.Worksheet consolidateSheet = (Excel.Worksheet)excelSheets.get_Item("Consolidated Data");

            Excel.Range rows = consolidateSheet.Rows;

            int count = rows.Count;

            foreach (KeyValuePair<string, Dictionary<string, string>> product in relationshipDictionary)
            {
                int productCount = 0;
                foreach (KeyValuePair<string, string> accessory in product.Value)
                {
                    string[,] values = new string[1,4];
                    values[0,0] = product.Key + "_[" + productCount + "]";
                    values[0,1] = product.Key;
                    values[0,2] = accessory.Key;
                    values[0,3] = accessory.Value;

                    rows.get_Range("A" + currentCount, "D" + currentCount).Value2 = values;

                    ++productCount;
                    ++currentCount;
                }
            }

            connection.Close();
        }

        private static DataTable GetWorksheetData(OleDbConnection oleConnection, string SheetName)
        {
            DataTable dt = new DataTable();
            OleDbDataAdapter odba = new OleDbDataAdapter(string.Format ("SELECT * FROM [{0}]",SheetName), oleConnection);
            odba.Fill(dt);
            return dt;
        }
    }
}