Adding row into DataTable

Question

How to add rows into datatable without foreach loop by reading records from csv?

var records = File.ReadLines(FD.FileName, Encoding.UTF8).Skip(1);
//not working because datatable is not thread safe
Parallel.ForEach(records, record => dataTable.Rows.Add(record.Split(',')));
//using for each is taking aroung 13 sec for 45000 records, need a better solution than this.
foreach (var record in records)
 {
 dataTable.Rows.Add(record.Split(','));
 }

Answer 1

Have you tried using a library like LINQ to CSV?

LINQ to CSV library

I have used it recently and its pretty good.

Basically you set the structure in your class like below

using LINQtoCSV;
using System;
class Product
{
    [CsvColumn(Name = "ProductName", FieldIndex = 1)]
    public string Name { get; set; }
    [CsvColumn(FieldIndex = 2, OutputFormat = "dd MMM HH:mm:ss")]
    public DateTime LaunchDate { get; set; }
    [CsvColumn(FieldIndex = 3, CanBeNull = false, OutputFormat = "C")]
    public decimal Price { get; set; }
    [CsvColumn(FieldIndex = 4)]
    public string Country { get; set; }
    [CsvColumn(FieldIndex = 5)]
    public string Description { get; set; }
}

There after use the below code to read the file

CsvFileDescription inputFileDescription = new CsvFileDescription
{
    SeparatorChar = ',', 
    FirstLineHasColumnNames = true
};
CsvContext cc = new CsvContext();
IEnumerable<Product> products =
    cc.Read<Product>("products.csv", inputFileDescription);
// Data is now available via variable products.
var productsByName =
    from p in products
    orderby p.Name
    select new { p.Name, p.LaunchDate, p.Price, p.Description };
// or ...
foreach (Product item in products) { .... }

Let me know how it goes

Answer 2

I'm sure, that you don't need to display 50000 of rows at once. Just because this is not human-readable.

Consider pagination: load first N rows, then, when user clicks "Next page" button, load next N rows, and so on.

Having this class for data row:

public class MyDataRow
{
    public int Id { get; set; }

    public string Name { get; set; }
}

pagination demo will be like this:

public partial class Form1 : Form
{
    private const int PageSize = 10;
    private readonly BindingList<MyDataRow> dataRows;
    private readonly IEnumerator<string> csvLinesEnumerator;

    public Form1()
    {
        InitializeComponent();

        // start enumeration
        csvLinesEnumerator = ReadLines().Skip(1).GetEnumerator();
        // assign data source to DGV
        dataRows = new BindingList<MyDataRow>();
        dataGridView1.DataSource = dataRows;
        // fetch first page
        FetchNextPage();
    }

    private void button1_Click(object sender, EventArgs e) => FetchNextPage();

    private void FetchNextPage()
    {
        // disable notifications
        dataRows.RaiseListChangedEvents = false;
        try
        {
            dataRows.Clear();

            while (dataRows.Count < PageSize)
            {
                if (csvLinesEnumerator.MoveNext())
                {
                    // parse next line and make row object from line's data
                    var lineItems = csvLinesEnumerator.Current.Split(',');
                    var row = new MyDataRow
                    {
                        Id = int.Parse(lineItems[0]),
                        Name = lineItems[1]
                    };

                    // add next row to the page
                    dataRows.Add(row);
                }
                else
                {
                    break;
                }
            }
        }
        finally
        {
            // enable notifications and reload data source into DGV
            dataRows.RaiseListChangedEvents = true;
            dataRows.ResetBindings();
        }
    }

    /// <summary>
    /// This is just File.ReadLines replacement for testing purposes
    /// </summary>
    private IEnumerable<string> ReadLines()
    {
        for (var i = 0; i < 50001; i++)
        {
            yield return $"{i},{Guid.NewGuid()}";
        }
    }
}

UI:

Another option is to use DataGridView.VirtualMode , but it mostly optimized for cases, when you know the total number of rows (eg, you can count them via SQL query), which isn't true for CSV files.