With C#, how can I convert a byte array of binary data into a custom-typed object that models the data?

Question

Scenario: I have received raw binary data via HTTP and stored the data into a byte array. I have documentation that describes various fields that the binary data can represent, but the actual meaning of the data must be determined at run-time. For example, if the byte that represents the occurrence of an error = 1, then the meaning of the next byte changes.

Using C# with .NET 4, I want to create one or more classes that mirror the fields described in the documentation and then somehow initialize the classes using the byte array of binary data. I would like the solution to minimize code duplication and also to be modular and elegant.

I have looked into creating Serializable classes, but I don't see how that can work since I am starting with a byte array that was not created (and therefore, not serialized) by me.

I have also attempted to use generics and reflection in order to retrieve the sizes and types of the fields contained in custom classes. I then used that information to dynamically slice out data from the byte array and assign it to the corresponding fields. However, this method resulted in a lot of ugly, unmanageable code.

Any advice or pointers on designing an extensible, decoupled solution for this problem would be much appreciated.

Edit: Example of classes containing fields that mirror the fields in the specification

public class PriceHistoryResponse : BinaryResponse
{
    public List<Quote> quotes { get; set; }
    private CountData countData { get; set; }
    private EndingDelimiterSection endingDelimiterSection { get; set; }

    /* This code performs the logic needed to check for optional fields
    and to find the number of times that certain fields are repeated */
    public PriceHistoryResponse(byte[] responseBytes) : base(responseBytes)
    {
        countData = new CountData();
        ParseResponseSection(countData);

        quotes = new List<Quote>();
        for (int i = 0; i < countData.quoteCount; i++)
        {
            quotes.Add(new Quote());

            quotes[i].symbolData = new SymbolData();
            ParseResponseSection(quotes[i].symbolData);

            if (quotes[i].symbolData.errorCode == 1)
            {
                quotes[i].errorData = new ErrorData();
                ParseResponseSection(quotes[i].errorData);
                break;
            }

            quotes[i].chartBarData = new ChartBarData();
            ParseResponseSection(quotes[i].chartBarData);

            quotes[i].chartBars = new List<ChartBar>();
            for (int j = 0; j < quotes[i].chartBarData.chartBarCount; j++)
            {
                quotes[i].chartBars.Add(new ChartBar());
                ParseResponseSection(quotes[i].chartBars[j]);
            }
        }

        endingDelimiterSection = new EndingDelimiterSection();
        ParseResponseSection(endingDelimiterSection);
    }
}

class CountData : IResponseSection
{
    public int quoteCount { get; set; }
}

public class Quote
{
    public SymbolData symbolData { get; set; }
    public ErrorData errorData { get; set; }
    public ChartBarData chartBarData { get; set; }
    public List<ChartBar> chartBars { get; set; }
}

public class SymbolData : IResponseSection
{
   public string symbol { get; set; }
   public byte errorCode { get; set; }
}

public class ErrorData : IResponseSection
{
    public string errorText { get; set; }
}

public class ChartBarData : IResponseSection
{
    public int chartBarCount { get; set; }
}

public class ChartBar : IResponseSection
{
    public float close { get; set; }
    public float high { get; set; }
    public float low { get; set; }
    public float open { get; set; }
    public float volume { get; set; }
    public long timestamp { get; set; }
}

Answer 1

You can try to do something like this:

public object ByteArrayToObject(byte[] byteArray)
{
    try
    {
        // convert byte array to memory stream
        System.IO.MemoryStream memoryStream = new System.IO.MemoryStream(byteArray);

        // create new BinaryFormatter
        System.Runtime.Serialization.Formatters.Binary.BinaryFormatter binaryFormatter
                        = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();

        // set memory stream position to starting point
        memoryStream.Position = 0;

        // Deserializes a stream into an object graph and return as a object.
        return binaryFormatter.Deserialize(memoryStream);
    }
    catch (Exception ex)
    {
        Console.WriteLine("Exception caught in process: {0}", ex.ToString());
    }

    return null;
}

EDIT

Take a look at protobuf-net. I think this is what you need:

http://code.google.com/p/protobuf-net/

protected bool EvaluateBuffer(byte[] buffer, int length)
{
    if (length < 8)
    {
        return false;
    }

    MessageType messageType = (MessageType)BitConverter.ToInt32(buffer, 0);
    int size = BitConverter.ToInt32(buffer, 4);
    if (length < size + 8)
    {
        return false;
    }

    using (MemoryStream memoryStream = new MemoryStream(buffer))
    {
        memoryStream.Seek(8, SeekOrigin.Begin);
        if (messageType == MessageType.MyMessage)
        {
            MyMessage message = 
                ProtoBuf.Serializer.Deserialize<MyMessage>(memoryStream);
        }
    }
}

Answer 2

There's a lot of project that will let you (de)serialize classes, but if the format is set and it's not to some standard like protocol buffers then I guess they won't help you much. Are you sure it's not some standard format?

If you need to read a byte to find out what the next bytes mean, my guess is you need to read them byte by byte using a BinaryReader and then pass the remainder of the byte array to a different function based on the result.

If you want to avoid having to handcode these 'parsers' then you could make a little DSL or something that would generate the dataclasses and accompanying code to read one from the byte array. But this sounds like doing a lot of work to save a little.

How many different classes do you need to be able to read from these byte arrays? My first guess would be to just write the code by hand. You'd need to have a lot of different classes or complicated logic to recuperate your development efforts.

Some more random ideas: look at the sourcecode of protocol buffer implementations, maybe you can pick ideas from those. http://code.google.com/p/protobuf-net/ or http://code.google.com/p/protobuf-csharp-port/

Rgds Gert-Jan

Answer 3

I pasted your code into VS, clicked 'Generate Method Stub' a few times and moved it around some. I guess this would do the trick.

The code you provide is pretty clever, it's a bit like a visitor pattern where overloading switches to the right method.

 public class BinaryResponse {

        private BinaryReader _rdr;
        public BinaryResponse(byte[] responseBytes) {
            _rdr = new BinaryReader(new MemoryStream(responseBytes)); // wrap the byte[] in a BinaryReader to be able to pop the bytes off the top
        }

        protected void ParseResponseSection(CountData countData) {
            countData.quoteCount = _rdr.ReadInt16(); // guessing 64.000 quotes should be enough in one response, the documentation will have the type      
        }

        protected void ParseResponseSection(SymbolData symbolData) {
            symbolData.errorCode = _rdr.ReadByte(); // depending on your format, where is the ErrorCOde in the byte[]? the symbol might be first

            int symbolLength = _rdr.ReadInt16(); // if it's not written by a .Net WriteString on the other end better to read this count yourelf
            symbolData.symbol = new string(_rdr.ReadChars(symbolLength)); // read the chars and put into string
        }

        protected void ParseResponseSection(ErrorData errorData) {
            int errorLenth = _rdr.ReadInt16();
            errorData.errorText = new string(_rdr.ReadChars(errorLenth));
        }

        protected void ParseResponseSection(ChartBarData chartBarData) {
            chartBarData.chartBarCount = _rdr.ReadInt16();
        }

        protected void ParseResponseSection(ChartBar chartBar) {
            // check the order with the documentation, also maybe some casting is needed because other types are in the byte[]
            chartBar.close = _rdr.ReadSingle();
            chartBar.high = _rdr.ReadSingle();
            chartBar.low = _rdr.ReadSingle();
            chartBar.open = _rdr.ReadSingle();
            chartBar.timestamp = _rdr.ReadInt64();
        }

        protected void ParseResponseSection(EndingDelimiterSection endingDelimiterSection) {
            int checkValue = _rdr.ReadInt16();
            if (checkValue != 12345) throw new InvalidDataException("Corrupt Response! Expecting End Delimiter"); // assert that the end delimiter is some value
        }
    }

Is this what you're looking for? You didn't say anything about encoding, you may need to take that into account when reading bytes etc.

Regards Gert-Jan