序列化/反序列化大数据集

Question

I have a reporting tool that sends query requests to a server. After the query is done by the server the result is sent back to the requesting reporting tool. The communication is done using WCF.

The queried data, stored in a DataSet object, is very large and is usually round about 100mb big.

To fasten the transmission I serialize (BinaryFormatter) and compress the DataSet.The transmitted object between the server and reporting tool is a byte array.

However after a few requests the reporting tool throws an OutOfMemoryException when it tries to deserialize the DataSet. The exception is thrown when I call:

dataSet = (DataSet) formatter.Deserialize(dstream);

dstream is the DeflateStream used to decompress the transmitted compressed byte array.

The exception occurs in a sub call of formatter.Deserialize when the byte array is created out of the stream.

Is there any other way of binary serialization that has a better mechanism to prevent this exception?

Implementation:

The method to serialize and compress the DataSet (used by the server)

public static byte[] Compress(DataSet dataSet)
{
    using (var input = new MemoryStream())
    {
        var binaryFormatter = new BinaryFormatter();
        binaryFormatter.Serialize(input, dataSet);

        using (var output = new MemoryStream())
        {
            using (var compressor = new DeflateStream(output, CompressionLevel.Optimal))
            {
                input.Position = 0;

                var buffer = new byte[1024];

                int read;

                while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
                    compressor.Write(buffer, 0, read);

                compressor.Close();

                return output.ToArray();
            }
        }
    }
}

The method to decompress and deserialize the DataSet (used by the reporting tool)

public static DataSet Decompress(byte[] data)
{
    DataSet dataSet;

    using (var input = new MemoryStream(data))
    {
        using (var dstream = new DeflateStream(input, CompressionMode.Decompress))
        {
            var formatter = new BinaryFormatter();
            dataSet = (DataSet) formatter.Deserialize(dstream);
        }
    }

    return dataSet;
}

Stacktrace:

at System.Array.InternalCreate(Void* elementType, Int32 rank, Int32* pLengths, Int32* pLowerBounds)
at System.Array.CreateInstance(Type elementType, Int32 length)
at System.Array.UnsafeCreateInstance(Type elementType, Int32 length)
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ParseArray(ParseRecord pr)
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ParseObject(ParseRecord pr)
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Parse(ParseRecord pr)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadArray(BinaryHeaderEnum binaryHeaderEnum)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream)
at DRX.PTClientMonitoring.Infrastructure.Helper.DataSetCompressor.Decompress(Byte[] data) in c:\_develop\PTClientMonitoringTool\PTClientMonitoringTool\Source\DRX.PTClientMonitoring.Infrastructure\Helper\DataSetCompressor.cs:line 51
at DRX.PTClientMonitoring.Reporting.ViewModels.ShellViewModel.<>c__DisplayClassf.<ExecudeDefinedQuery>b__4() in c:\_develop\PTClientMonitoringTool\PTClientMonitoringTool\Source\DRX.PTClientMonitoring.Reporting\ViewModels\ShellViewModel.cs:line 347

Answer 1

Before serializing, set:

yourDataSet.RemotingFormat = SerializationFormat.Binary;

That should help a lot. The default even when using BinaryFormatter is xml.

Note, however, that DataSet and DataTable are inherently not great candidates for optimization. There are a lot of great serialization tools that will do a much better job of packing your data, but they invariable require strong type models, ie List<SomeSpecificType> where SomeSpecificType is a POCO/DTO class. Even WCF only barely tolerates DataTable / DataSet . So if you can get rid of your dependency on DataTable / DataSet : I strongly advise doing so.

Another option is to send the data as a Stream . I'm pretty sure WCF supports this natively, but this would in theory allow you to have a different Stream ( not MemoryStream ) that is actually much larger. As a cheap option you could use a temporary file as a scratch area, but if that works you could investigate a custom in-memory stream that stitches multiple buffers together.