How to import large amounts of data from CSV file to DataGridView efficiently

Question

I have 300 csv files that each file contain 18000 rows and 27 columns.

Now, I want to make a windows form application which import them and show in a datagridview and do some mathematical operation later.

But, my performance is very inefficiently...

After search this problem by google, I found a solution "A Fast CSV Reader". ( http://www.codeproject.com/Articles/9258/A-Fast-CSV-Reader )

I'm follow the code step by step, but my datagridview still empty.

I don't know how to solve this problem.

Could anyone tell me how to do or give me another better way to read csv efficiently.

Here is my code...

using System.IO;
using LumenWorks.Framework.IO.Csv;

private void Form1_Load(object sender, EventArgs e)
{
    ReadCsv();
}

void ReadCsv()
{
    // open the file "data.csv" which is a CSV file with headers
    using (CachedCsvReader csv = new
           CachedCsvReader(new StreamReader("data.csv"), true))
    {
        // Field headers will automatically be used as column names
        dataGridView1.DataSource = csv;
    }
}

Here is my input data: https://dl.dropboxusercontent.com/u/28540219/20130102.csv

Thanks...

Answer 1

You can also do like

  private void ReadCsv()
    {
        string filePath = @"C:\..\20130102.csv";
        FileStream fileStream = null;
        try
        {
            fileStream = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
        }
        catch (Exception ex)
        { 
            return;
        }

        DataTable table = new DataTable();
        bool isColumnCreated = false;
        using (StringReader reader = new StringReader(new StreamReader(fileStream, Encoding.Default).ReadToEnd()))
        {
            while (reader.Peek() != -1)
            {
                string line = reader.ReadLine();

                if (line == null || line.Length == 0)
                    continue; 

                string[] values = line.Split(',');

                if(!isColumnCreated)
                {
                    for(int i=0; i < values.Count(); i++)
                    {
                        table.Columns.Add("Column" + i);
                    }
                    isColumnCreated = true;
                }

                DataRow row = table.NewRow();

                for(int i=0; i < values.Count(); i++)
                {
                    row[i] = values[i];
                }
                table.Rows.Add(row);
            }
        }

        dataGridView1.DataSource = table;
    }

Based on you performance requirement, this code can be improvised. It is just a working sample for your reference.

I hope this will give some idea.

Answer 2

The data you provide contains no headers (first line is a data line). So I got an ArgumentException (item with same key added) when I tried to add the csv reader to the DataSource. Setting the hasHeaders parameter in the CachCsvReader constructor did the trick and it added the data to the DataGridView (very fast).

using (CachedCsvReader csv = new CachedCsvReader(new StreamReader("data.csv"), false))
{
     dataGridView.DataSource = csv;
}

Hope this helps!