无法从SQL Server和C＃应用程序在csv中下载具有80列的6到700万条记录

Question

I need to download an extract from my application created on asp.net C#. It reads from SQL Server 2008 and tries to write it into .csv file. There are 6 to 7 million rows with approx 80 columns. The application breaks and stops responding and I'm not able to download through web browser.

As an alternative I tried creating a windows service which runs on server to extract the data and create a csv file. But in case of multiple download request the server runs out of memory and the service breaks and stops working.

Total data in table is 700 MB and to read this the server memory reaches utilization of 5 GB out 8 GB for each request. In case if there are multiple async request the memory utilization goes to 100% and service stops and file is not generated.

Currently using data table and rows as follows, need a way to download the data using C# application on request or any other means which can generate the csv file from the data given from the stored procedure.

Thanks

// the method to extract data
public getdata( xyz ....)
{
    dbInstance = ConfigDBDAL.GetTransactionDBInstance(userDO);

    DbCommand dbCommand = dbInstance.GetStoredProcCommand(PR_GET_EXTRACTS);
    dbCommand.CommandTimeout = 0;

    // a few parameter passing like this
    DbParameter paramStartDate = new SqlParameter("@StartDate", typeof(DateTime));
    paramStartDate.Value = startDate;
    dbCommand.Parameters.Add(paramStartDate);

    DbParameter paramStartDate = new SqlParameter("@EndDate", typeof(DateTime));
    paramStartDate.Value = startDate;
    dbCommand.Parameters.Add(paramStartDate);

    // Creating a data table for the required 80 columns 
    datatable = new DataTable();

    datatable.Columns.Add("COL1");
    datatable.Columns.Add("COL2");
    datatable.Columns.Add("COL3");
    .
    .
    .
    datatable.Columns.Add("COL80");

    using (IDataReader reader = dbInstance.ExecuteReader(dbCommand))
    {
        DataRow newRow;

        while (reader.Read())
        {
            row = datatable.NewRow();
            row["COL1"] = reader["COL1"];
            row["COL2"] = reader["COL2"];
            .
            .
            row["COL80"] = reader["COL80"];

            datatable.Rows.Add(row);
        }
    }

    return datatable;
}

Answer 1

DataTable class produces some overhead. Try to build pure CSV string using StringBuilder :

var sb = new StringBuilder();
sb.AppendLine("sep=,");
sb.AppendLine("COL1,COL2,COL3,...,COL-N"); // columns

using (IDataReader reader = dbInstance.ExecuteReader(dbCommand))
{
    DataRow newRow;
    while (reader.Read())
    {
        sb.Append((string)reader["COL1"] + ",");
        sb.Append((string)reader["COL2"] + ",");
        sb.Append((string)reader["COL3"] + ",");
        ...
        sb.Append((string)reader["COL-N"] + Environment.NewLine);
    }
}

return sb.ToString(); // returns string representing CSV file content

Answer 2

You can also try this method.

public static void ExportPlantData(string channelId)
{
    string query = string.Empty;

    DataService dataService = new DataService();
    DbCommand dataCmd = null;
    DataTable contentToExport = new DataTable();

    try
    {
        query = "SELECT * from tablename";

        dataCmd = dataService.Database.GetSqlStringCommand(query);
        contentToExport = dataService.ExecuteDataTable(dataCmd);
        ExportToCSV(contentToExport);
    }    
}


public static void ExportToCSV(DataTable contentToexport)
{
    StringBuilder csvData = new StringBuilder();
    StringBuilder headers = new StringBuilder();

    foreach (DataRow row in contentToexport.Rows)
    {
        headers = string.Empty;
        foreach (DataColumn column in contentToexport.Columns)
        {
            csvData.Append(row[column].ToString() + ",");
            headers.Append(column.ColumnName + ",");
        }

       csvData.Append("\r\n");
       headers.Append("\r\n");
    }

    string contentToExport = headers.Append(csvData.ToString()).ToString();
    string attachment = "attachment; filename=export.csv";

    HttpContext.Current.Response.Clear();
    HttpContext.Current.Response.ClearHeaders();
    HttpContext.Current.Response.ClearContent();
    HttpContext.Current.Response.AddHeader("content-disposition", attachment);
    HttpContext.Current.Response.ContentType = "application/csv";
    HttpContext.Current.Response.AddHeader("Pragma", "public");
    HttpContext.Current.Response.Write(contentToExport);
    System.Web.HttpContext.Current.ApplicationInstance.CompleteRequest();
}