Efficiently writing an int array to file

Question

I have a potentially larger int array that I'm writing to a file using BinaryWriter . Of course, I can use the default approach.

using (BinaryWriter writer = new BinaryWriter(File.Open(path, FileMode.Create)))
{
    writer.Write(myIntArray.Length);
    foreach (int value in myIntArray)
        writer.Write(value);
}

But this seems horribly inefficient. I'm pretty sure an int array stores data contiguously in memory. Is there no way to just write the memory directly to a file like you can with a byte array? Maybe a way to cast (not copy) the int array to a byte array?

Answer 1

I thought it would be interesting to benchmark each of the methods outlined above, the original from @Jonathan-Wood (TestCopyStream), the Span suggestion from @Mike-Zboray (TestCopySpan) and the Buffer BlockCopy from @oleg-bondarenko (TestCopySpanByteCopy) [yup, naming things is hard].

I'm generating int arrays of size N of random numbers, the same set for each run.

Here's the results:

|               Method |     N |     Mean |     Error |    StdDev |   Median | Ratio | RatioSD | Rank |   Gen 0 | Gen 1 | Gen 2 | Allocated |
|--------------------- |------ |---------:|----------:|----------:|---------:|------:|--------:|-----:|--------:|------:|------:|----------:|
|         TestCopySpan |  1000 | 1.372 ms | 0.0382 ms | 0.1109 ms | 1.348 ms |  1.00 |    0.11 |    1 |       - |     - |     - |    4984 B |
|       TestCopyStream |  1000 | 1.377 ms | 0.0324 ms | 0.0935 ms | 1.364 ms |  1.00 |    0.00 |    1 |       - |     - |     - |    4984 B |
| TestCopySpanByteCopy |  1000 | 2.215 ms | 0.0700 ms | 0.2008 ms | 2.111 ms |  1.62 |    0.19 |    2 |  3.9063 |     - |     - |   13424 B |
|                      |       |          |           |           |          |       |         |      |         |       |       |           |
|         TestCopySpan | 10000 | 1.617 ms | 0.1167 ms | 0.3155 ms | 1.547 ms |  0.80 |    0.19 |    1 |       - |     - |     - |     864 B |
|       TestCopyStream | 10000 | 2.032 ms | 0.0776 ms | 0.2251 ms | 1.967 ms |  1.00 |    0.00 |    2 |       - |     - |     - |    4984 B |
| TestCopySpanByteCopy | 10000 | 2.433 ms | 0.0703 ms | 0.2040 ms | 2.430 ms |  1.21 |    0.18 |    3 | 11.7188 |     - |     - |   45304 B |

Answer 2

There is support for the most efficient form without any copying in .NET Core via MemoryMarshal.Cast and Span<T> . This directly reinterprets the memory but this is potentially nonportable across platforms so it should be used with care:

 int[] values = { 1, 2, 3 };

 using (var writer = new BinaryWriter(File.Open(path, FileMode.Create)))
 {
     Span<byte> bytes = MemoryMarshal.Cast<int, byte>(values.AsSpan());
     writer.Write(bytes);
 }

Some relevant discussion of this API when it was moved from MemoryExtensions.NonPortableCast

However I will say your original is actually going to be fairly efficient because both BinaryWriter and FileStream have their own internal buffers that are used when writing ints like that.

Answer 3

I am not sure but you could try myIntArray.SelectMany(BitConverter.GetBytes).ToArray() (execution time 4700ms)

there is other approach

var binFormatter = new BinaryFormatter();
var mStream = new MemoryStream();
binFormatter.Serialize(mStream, myIntArray);  

mStream.ToArray(); 

(execution time 2700ms)

This is the fastest approach that I have found and checked with watcher profiler: execution time 1500 ms (no 3d party components), other ways are about 1700ms (MemoryMarshal) , "for each" method -2700 ms

            int maxValue = Int32.MaxValue / 50;
            int[] myIntArray = Enumerable.Range(0, maxValue).ToArray();
            var path = "e:\\temp\\1.test";
            using (BinaryWriter writer = new BinaryWriter(File.Open(path, FileMode.Create)))
            {
                int intLength = myIntArray.Length;
                writer.Write(intLength);

                byte[] bytes = new byte[intLength * sizeof(int)];
                Buffer.BlockCopy(myIntArray, 0, bytes, 0, sizeof(byte));
                writer.Write(bytes);
            }