没有结构元数据本身的C＃序列化

Question

Right now I'm working on a game engine. To be more efficient and keep data from the end user, I'm trying to use serialization on a modified form of the Wavefront's *.OBJ format. I have multiple structs set up to represent data, and the serialization of the objects works fine except it takes up a significant amount of file space (at least x5 that of the original OBJ file).

To be specific, here's a quick example of what the final object would be (in a JSON-esque format):

{
    [{float 5.0, float 2.0, float 1.0}, {float 7.0, float 2.0, float 1.0}, ...]
    // ^^^ vertex positions
    // other similar structures for colors, normals, texture coordinates
    // ...

    [[{int 1, int 1, int 1}, {int 2, int 2, int 1}, {int 3, int 3, int 2}], ...]
    //represents one face; represents the following
    //face[vertex{position index, text coords index, normal index}, vertex{}...]
}

Basically, my main issue with the method of serializing data (binary format) is it saves the names of the structs, not the values. I'd love to keep the data in the format I have already, just without saving the struct itself in my data. I want to save something similar to the above, yet it'll still let me recompile with a different struct name later.

Here's the main object I'm serializing and saving to a file:

[Serializable()] //the included structs have this applied
public struct InstantGameworksObjectData
{
    public Position[] Positions;
    public TextureCoordinates[] TextureCoordinates;
    public Position[] Normals;

    public Face[] Faces;
}

Here's the method in which I serialize and save the data:

IFormatter formatter = new BinaryFormatter();

long Beginning = DateTime.Now.Ticks / 10000000;
foreach (string file in fileNames)
{
    Console.WriteLine("Begin " + Path.GetFileName(file));

    var output = InstantGameworksObject.ConvertOBJToIGWO(File.ReadAllLines(file));

    Console.WriteLine("Writing file");

    Stream fileOutputStream = new FileStream(outputPath + @"\" + Path.GetFileNameWithoutExtension(file) + ".igwo", FileMode.Create, FileAccess.Write, FileShare.None);
    formatter.Serialize(fileOutputStream, output);


    Console.WriteLine(outputPath + @"\" + Path.GetFileNameWithoutExtension(file) + ".igwo");
}

The output, of course, is in binary/hex (based on what program you use to view the file), and that's great:

But putting it into a hex-to-text converter online yields specific name data:

In the long run, this could mean gigabytes worth of useless data. How can I save my C# object with the data in the correct format, just without the extra meta-clutter?

Answer 1

As you correctly note, the standard framework binary formatters include a host of metadata about the structure of the data. This is to try to keep the serialised data self-describing. If they were to separate the data from all that metadata, then the smallest change to the structure of classes would render the previously serialised data useless. By that token, I doubt you'd find any standard framework method of serialising binary data that didn't include all the metadata.

Even ProtoBuf includes the semantics of the data in the file data, albeit with less overhead.

Given that the structure of your data follows the reasonably common and well established form of 3D object data, you could roll your own format for your assets which strips the semantics and only stores the raw data. You can implement read and write methods easily using the BinaryReader/BinaryWriter classes (which would be my preference). If you're looking to obfuscate data from the end user, there are a variety of different ways that you could achieve that with this approach.

For example:

public static InstantGameworksObjectData ReadIgoObjct(BinaryReader pReader)
{
    var lOutput = new InstantGameworksObjectData();

    int lVersion = pReader.ReadInt32();     // Useful in case you ever want to change the format

    int lPositionCount = pReader.ReadInt32();   // Store the length of the Position array before the data so you can pre-allocate the array.
    lOutput.Positions = new Position[lPositionCount];
    for ( int lPositionIndex = 0 ; lPositionIndex < lPositionCount ; ++ lPositionIndex )
    {
        lOutput.Positions[lPositionIndex] = new Position();
        lOutput.Positions[lPositionIndex].X = pReader.ReadSingle();
        lOutput.Positions[lPositionIndex].Y = pReader.ReadSingle();
        lOutput.Positions[lPositionIndex].Z = pReader.ReadSingle();
        // or if you prefer...  lOutput.Positions[lPositionIndex] = Position.ReadPosition(pReader);
    }

    int lTextureCoordinateCount = pReader.ReadInt32();
    lOutput.TextureCoordinates = new TextureCoordinate[lPositionCount];
    for ( int lTextureCoordinateIndex = 0 ; lTextureCoordinateIndex < lTextureCoordinateCount ; ++ lTextureCoordinateIndex )
    {
        lOutput.TextureCoordinates[lTextureCoordinateIndex] = new TextureCoordinate();
        lOutput.TextureCoordinates[lTextureCoordinateIndex].X = pReader.ReadSingle();
        lOutput.TextureCoordinates[lTextureCoordinateIndex].Y = pReader.ReadSingle();
        lOutput.TextureCoordinates[lTextureCoordinateIndex].Z = pReader.ReadSingle();
        // or if you prefer...  lOutput.TextureCoordinates[lTextureCoordinateIndex] = TextureCoordinate.ReadTextureCoordinate(pReader);
    }

    // ...
}

As far as space efficiency and speed goes, this approach is hard to beat. However, this works well for the 3D objects as they're fairly well-defined and the format is not likely to change, but this approach may not extend well to the other assets that you want to store.

If you find you are needing to change class structures frequently, you may find you have to write lots of if-blocks based on version to correctly read a file, and have to regularly debug issues where the data in the file is not quite in the format you expect. A happy medium might be to use something such as ProtoBuf for the bulk of your development until you're happy with the structure of your data object classes, and then writing raw binary Read/Write methods for each of them before you release.

I'd also recommend some Unit Tests to ensure that your Read and Write methods are correctly persisting the object to avoid pulling your hair out later.

Hope this helps