How to load large amount of data to DataTable

Question

I am exporting data from Excel to a DataTable, but I am getting some performance issues when my Excel file contains large amount of rows...

public DataView LoadFromExcel()
{
    Microsoft.Office.Interop.Excel.Application application = 
                           new Microsoft.Office.Interop.Excel.Application();
    Workbook workbook = null;
    Worksheet worksheet = null;
    string filename = null;
    OpenFileDialog file = new OpenFileDialog();
    if (true == file.ShowDialog())
    {
        filename = file.FileName;
    }
    workbook = application.Workbooks.Open(filename, true, true);
    worksheet = workbook.Sheets[1];
    Range range = worksheet.UsedRange;
    int row = range.Rows.Count;
    int columns = range.Columns.Count;
    System.Data.DataTable dt = new System.Data.DataTable();
    for (int i = 1; i <= columns; i++)
    {
        dt.Columns.Add((range.Cells[1, i] as Range).Value2.ToString());
    }
    for (row = 2; row <= range.Rows.Count; row++)
    {
        DataRow dr = dt.NewRow();
        for (int column = 1; column <= range.Columns.Count; column++)
        {
            dr[column - 1] = (range.Cells[row, column] as
                    Microsoft.Office.Interop.Excel.Range).Value2.ToString();
        }
        dt.Rows.Add(dr);
        dt.AcceptChanges();
    }
    workbook.Close(true, Missing.Value, Missing.Value);
    application.Quit();
    return dt.DefaultView;
}

Is there any way I can solve this problem? Please help.

Answer 1

You can do it with the help of OLEDb provider. I have tried doing for 50000 records. It may help you, just try below code:

        // txtPath.Text is the path to the excel file.
        string conString = @"Provider=Microsoft.ACE.OLEDB.12.0;" + "Data Source=" + txtPath.Text + ";" + "Extended Properties=" + "\"" + "Excel 12.0;HDR=YES;" + "\"";

        OleDbConnection oleCon = new OleDbConnection(conString);

        OleDbCommand oleCmd = new OleDbCommand("SELECT field1, field2, field3 FROM [Sheet1$]", oleCon);

        DataTable dt = new DataTable();

        oleCon.Open();
        dt.Load(oleCmd.ExecuteReader());
        oleCon.Close();

You have to take care of few things:

1. Name of the sheet should be Sheet1 or else give the proper name in the query.
2. While reading the sheet, sheet should not be open.
3. The column name should be properly defined in the query
4. Column name should be on the first row in the sheet

I hope it will help you... Let me know if any thing more you require... :)

Answer 2

您可以使用Sql批量复制来执行此类操作。

Answer 3

I think this is not the right approach.

For inserting large amount of data into a table, you should use "Bulk Insert" feature of your database and during bulk insert, you should turn off the database log and roll-back features. Otherwise the bulk insert would act just like bunch of ordinary inserts.

I know Oracle and SQL Server has this feature and some NoSQL databases has it too. Since you have not mentioned what is your database, it helps to google it.

Answer 4

Try reading the values to variables and do some filters in order to avoid sending wrong values that can affect your database. It is wrong to save unknown data to database most expecially MS SQL - do some filtering to make the saving easier and preserve your DB health..