如何高效地将大型二维数组写入 CSV 文件

Question

EDIT: Using the solution provided, I was able to get it to work (not the most efficient but it works) using the following:

    public static void WriteArray(object[,] dataArray, string filePath, bool deleteFileIfExists = true)
    {
        if (deleteFileIfExists && File.Exists(filePath))
        {
            File.Delete(filePath);
        }

        byte columnCount = (byte) dataArray.GetLength(1);
        int rowCount = dataArray.GetLength(0);

        for(int row = ReadExcel.ExcelIndex; row <= rowCount; row++)
        {
            List<string> stringList = new List<string>();
            for(byte column = ReadExcel.ExcelIndex; column <= columnCount; column++)
            {
                stringList.Add(dataArray[row, column].ToString());
            }
            string rowAsString = StringifyRow(stringList.ToArray());
            WriteLine(filePath, rowAsString);
        }
    }

    public static string StringifyRow(string[] row)
    {
        return string.Join(",", row);
    }

    public static void WriteLine(string filePath, string rowAsString)
    {
        using (StreamWriter writer = new StreamWriter(filePath, append: true))
        {
            writer.WriteLine(rowAsString);
            writer.Flush();
        }
    }

I am working on a data consolidation application. This app will read in data as 2D arrays, manipulate some of the columns in memory, consolidate arrays, then write to a csv file.

I am working on a test method of writing to CSV. Currently my array is just over 70K rows and 29 columns. In production, it will always be at least 4 times as many rows. For the test, I did not consolidate any arrays. This is just from one data source.

In order to write to CSV, I have tried numerous methods:

1. Pass the array (object[,]) to a helper method that loops through each row, creates a string from the values, then appends the string to a list, finally passes the list to the csvwriter to write the list of strings to a file. I get OutOfMemory errors when creating the list of strings.
2. The same as above but with StringBuilder and no List, just a giant string with stringbuilder (found on .netperls) - same error - OutofMemory exception
3. No helper method, just pass the array to the csvwriter method, loop through each row, build a string (I've tried this with string builder as well) and write each line one by one to the file. I get an OutOfMemory exception.

Some of my code (some various methods have been commented out) is below:

using System.IO;
using System.Text;
using System.Collections.Generic;
namespace ExcelIO
{
    public static class CsvWriter
    {
        public static void WriteStringListCsv(List<string> data, string filePath, bool deleteIfExists = true)
        {
            if (deleteIfExists && File.Exists(filePath))
            {
                File.Delete(filePath);
            }

            //foreach(string record in data)
            //{
            //    File.WriteAllText(filePath, record);
            //}

            using (StreamWriter outfile = new StreamWriter(filePath))
            {
                foreach (string record in data)
                {
                    //trying to write data to csv
                    outfile.WriteLine(record);
                }
            }
        }

        public static void WriteDataArrayToCsv(List<string> data, string filePath, bool deleteIfExists = true)
        {
            if (deleteIfExists && File.Exists(filePath))
            {
                File.Delete(filePath);
            }

            //foreach(string record in data)
            //{
            //    File.WriteAllText(filePath, record);
            //}

            //using (StreamWriter outfile = new StreamWriter(filePath))
            //{
            //    File.WriteAllText(filePath, )
            //}

            using (StreamWriter outfile = new StreamWriter(filePath))
            {
                foreach(string record in data)
                {
                    outfile.WriteLine(record);
                }
            }
        }


        public static List<string> ConvertArrayToStringList(object[,] dataArray, char delimiter=',', bool includeHeaders = true)
        {
            List<string> dataList = new List<string>();
            byte colCount = (byte)dataArray.GetLength(1);
            int rowCount = dataArray.GetLength(0);

            int startingIndex = includeHeaders ? ReadExcel.ExcelIndex : ReadExcel.ExcelIndex + 1;

            //StringBuilder dataAsString = new StringBuilder();
            for (int rowIndex = startingIndex; rowIndex <= rowCount; rowCount++)
            {
                StringBuilder rowAsString = new StringBuilder();
                //string rowAsString = "";
                for (byte colIndex = ReadExcel.ExcelIndex; colIndex <= colCount; colIndex++)
                {

                    //rowAsString += dataArray[rowIndex, colIndex];
                    //rowAsString += (colIndex == colCount) ? "" : delimiter.ToString();
                    // Wrap in Quotes
                    rowAsString.Append($"\"{dataArray[rowIndex, colIndex]}\"");
                    if (colIndex == colCount)
                    {
                        rowAsString.Append(delimiter.ToString());
                    }
                }
                // Move to nextLine
                //dataAsString.AppendLine();
                //outfile.WriteLine(rowAsString);
                dataList.Add(rowAsString.ToString());
            }
            //return dataAsString.ToString();
            return dataList;
        }
    }
}

I've tried everything I've seen from searching online but everything gives me an OutOfMemroy exception when piecing together the rows (even just doing one row at a time and writing that). Is there a better way to efficiently and hopefully quickly write a large 2D array to a csv file?

Any and all tips are greatly appreciated.

Answer 1

With code like below I've never had any memory issues, even with huge data

    class Program
    {
        static void Main(string[] args)
        {
            StreamWriter writer = new StreamWriter("Filename");
            List<Record> data = new List<Record>();

            foreach (Record record in data)
            {
                string line = string.Join(",", record.field1, record.field2, record.field3, record.field4, record.field5);
                writer.WriteLine(line);
            }
            writer.Flush();
            writer.Close();
        }
    }
    public class Record
    {
        public string field1 { get; set; }
        public string field2 { get; set; }
        public string field3 { get; set; }
        public string field4 { get; set; }
        public string field5 { get; set; }

    }