如何在C＃中调用IEnumerable Datatable集合

Question

我正在尝试分批读取CSV文件，以避免在导入大型CSV文件时出现内存不足异常问题。

我正在使用IEnumerable Collection方法来从csv中分块读取数据并将其返回给另一个函数。

但是我将System.Collection.Generic.IEnumerable转换为System.Data.Datatable。 隐式转换是不可能的。

下面是代码：

    public void ImportData(string targetPathwithName, System.Data.DataTable dt1, string targetName)
    {
        var con = ConfigurationManager.ConnectionStrings["con1"].ConnectionString.ToString();
        var connection = new SqlConnection(con);



        dt1 =  GetFileData(targetPathwithName, dt1);

        connection.Open();

        var bulkCopy = new SqlBulkCopy(connection);
        bulkCopy.DestinationTableName = "[" + targetName + "]";
        bulkCopy.WriteToServer(dt1);
        bulkCopy.Close();
        connection.Close();

    }

public static IEnumerable<System.Data.DataTable> GetFileData(string sourceFileFullName,  System.Data.DataTable dt1)
    {            
        var con = ConfigurationManager.ConnectionStrings["con1"].ConnectionString.ToString();
        var connection = new SqlConnection(con);

        int chunkRowCount = 0;
        int RowCount = 0;

        using (var sr = new StreamReader(sourceFileFullName))
        {
            if (RowCount != 0)
            {
                string Row = null;
                //Read and display lines from the file until the end of the file is reached.                
                while ((Row = sr.ReadLine()) != null)
                {
                    chunkRowCount++;
                    //var chunkDataTable = ; //Code for filling datatable or whatever  
                    dt1.Rows.Add();

                    if (chunkRowCount == 10000)
                    {
                        chunkRowCount = 0;
                        yield return dt1;
                        dt1 = null;
                    }

                    else
                    {
                        int i = 0;

                        foreach (string Cell in Row.Split(','))
                        {
                            if (String.IsNullOrEmpty(Cell))
                            {
                                dt1.Rows[dt1.Rows.Count - 1][i] = DBNull.Value;
                                i = i + 1;
                            }
                            else if (Cell == "00.00.0000")
                            {
                                dt1.Rows[dt1.Rows.Count - 1][i] = DBNull.Value;
                                i = i + 1;
                            }
                            else
                            {
                                dt1.Rows[dt1.Rows.Count - 1][i] = Cell;
                                i = i + 1;
                            }
                        }
                    }

                }
            }
            RowCount = RowCount + 1;
        }
            //return last set of data which less then chunk size
            if (null != dt1)                           
                yield return dt1;            
        }

如何在另一个函数中调用类型为System.Data.Datatable的IEnumerable Collection？

Answer 1

实际上，您仅返回单个数据表，而不返回IEnumerable。

您可以做的是1）在函数GetFileData中创建IEnumerable的单个实例，例如IEnumerable DTables = new IEnumerable（）;。 2）每次为每个数据块创建一个新的datatable实例，并将该数据表添加到上面定义的IEnumerableCollection中，然后返回Collection。 3）用于函数ImportData中的每个循环以读取集合中的所有数据表。

Answer 2

您必须使用foreach循环， foreach代码进行如下修改：

public static void ImportData(string targetPathwithName, System.Data.DataTable dt1, string targetName)
{
    var con = ConfigurationManager.ConnectionStrings["con1"].ConnectionString.ToString();
    var connection = new SqlConnection(con);

    connection.Open();
    var bulkCopy = new SqlBulkCopy(connection);
    foreach (System.Data.DataTable dt in GetFileData(targetPathwithName, dt1))
    {
        bulkCopy.DestinationTableName = "[" + targetName + "]";
        bulkCopy.WriteToServer(dt);
    }
    bulkCopy.Close();
    connection.Close();
}

public static IEnumerable<System.Data.DataTable> GetFileData(string sourceFileFullName, System.Data.DataTable dt1)
{
    var con = ConfigurationManager.ConnectionStrings["con1"].ConnectionString.ToString();
    var connection = new SqlConnection(con);

    int chunkRowCount = 0;
    //int RowCount = 0;
    string Row;

    using (var sr = new StreamReader(sourceFileFullName))
    {
        //if (RowCount != 0) { //this is not meaningful here

        //Read and display lines from the file until the end of the file is reached.                
        while ((Row = sr.ReadLine()) != null)
        {
            chunkRowCount++;
            //var chunkDataTable = ; //Code for filling datatable or whatever  
            dt1.Rows.Add();

            int i = 0;

            foreach (string Cell in Row.Split(','))
            {
                if (String.IsNullOrEmpty(Cell))
                {
                    dt1.Rows[dt1.Rows.Count - 1][i] = DBNull.Value;
                    i = i + 1;
                }
                else if (Cell == "00.00.0000")
                {
                    dt1.Rows[dt1.Rows.Count - 1][i] = DBNull.Value;
                    i = i + 1;
                }
                else
                {
                    dt1.Rows[dt1.Rows.Count - 1][i] = Cell;
                    i = i + 1;
                }
            }

            if (chunkRowCount == 10000)
            {
                chunkRowCount = 0;
                yield return dt1;
                dt1.Clear(); // = null;
            }

        } //end while

        //}
        //RowCount = RowCount + 1;
    }

    //return last set of data which less then chunk size
    if (dt1.Rows.Count > 0)
        yield return dt1;
}

我已经测试过了，效果很好