Is it possible to write a large array of struct directly to a binary file in c#?

Question

I know there are lots of similar questions around on SO. But none of them had the answer I was looking for.

I need to write a very large array (about 1 GiB in storage) to a binary file. It consists of very simple structs (struct with just 4 readonly byte members).

Can I write that array in one go into a binary file/stream without copying the whole array into a byte array? I am bound to .NET Standard 2.0, so no MemoryMarshal to the help.

Cheers, Georg

Answer 1

Is this help you? copy by pointer

public static byte[] GetBytes<T>(T[] structAry) where T : struct
    {
        int len = (int)structAry.Length;
        int size = Marshal.SizeOf(default(T));
        byte[] arr = new byte[size * len];

        IntPtr ptr = Marshal.AllocHGlobal(size);
        try
        {
            for (int i = 0; i < len; ++i)
            {
                Marshal.StructureToPtr(structAry[i], ptr, true);
                Marshal.Copy(ptr, arr, i * size, size);
            }
        }
        catch (Exception e)
        {
            throw e;
        }
        finally
        {
            Marshal.FreeHGlobal(ptr);
        }
        return arr;
    }

Answer 2

It depends on what your scenario is, I will outline a few scenarios so you can pick the one that suits your case the most.

Your array is already in memory

You will not need to copy it into another array, you will be able to write it out.

You have a file containing the content you want to save into another file

You can use [File.Copy][1] in order to copy a file.

You cannot directly copy the data

If your data is to be extracted from a remote source or your data needs to be parsed/modified for some reason before saving, then you will need to load it into memory, there is no way around that. However, you can load smaller packets instead of loading the whole thing at once with the algorithm of

while (packet <- packets.Next) do
    initialize(packet)
    save(packet)
    finalize(packet)
while end

See more here: https://social.msdn.microsoft.com/Forums/vstudio/en-US/54eb346f-f979-49fb-aa2d-44dddad066bd/how-to-read-file-in-chunks-with-for-loop?forum=netfxbcl [1]: https://docs.microsoft.com/en-us/dotnet/api/system.io.file.copy?view=net-5.0

Answer 3

I don't know of a way of pointing to a chunk of memory in a way that you can use it with File.Write... without copying data.

I would suggest doing the writing in chunks:

1. Copy a part to a buffer
2. Write buffer to file
3. Copy next chunk
4. Write buffer to file
5. Repeat until whole object done.