通过C＃WCF REST Web服务作为MemoryStream构建和发送非常大的CSV数据集

Question

This topic is related to the question here:

How to send a CSV file using C# WCF RESTful (ie Web) Service?

I was able to send a very small data set by utilizing the method discussed at http://blogs.msdn.com/b/carlosfigueira/archive/2008/04/17/wcf-raw-programming-model-web.aspx , which explains how to utilize a "RAW" programming model to send the response -- I'm using the MemoryStream class.

public Stream ReturnCSVSample()
{
    string csv = "Hello, World\nGoodBye, For, Now\n";
    byte[] csvBytes = Encoding.UTF8.GetBytes(csv);

    WebOperationContext.Current.OutgoingResponse.ContentType = "text/csv";

    return new MemoryStream(csvBytes);
}

This works fine. However, in reality, I'm retrieving about 144,000 records from a SQL database table, two columns wide, and using string concatenation to format it into CSV. Then, I take this string,encode it as a bytes array and return it as a MemoryStream object. I let the service run for over 20 minutes and it still does not finish! I've determined the bottleneck is the string concatenation. Why? Because I performed the operation of retrieving these 144,000 records and storing it to a class I created and it took a few minutes, at most, to complete.

Either a string that large is inefficient or there is a better method to perform string concatenation for the amount of records I specified.

How can I send a CSV stream over a web service with such as a large data set?

Thank you.

Answer 1

A StreamWriter() would do a better job than string concatenation:

public Stream LoadStreamSample(Stream ms)
{
    using (StreamWriter sw = new StreamWriter(ms, Encoding.UTF8)){
        // Call SQL and iterate over rows from SqlDataReader here...
        while (dr.Read()){
            // replace following with csv of one record
            string csv = "Hello, World\nGoodBye, For, Now\n";
            sw.WriteLine(csv)
        }
    }
    return ms;
}