包含char数组时快速读取C结构

Question

我有以下C结构

struct MyStruct {
    char chArray[96];
    __int64 offset;
    unsigned count;
}

我现在有一堆用C创建的文件，其中包含数千种结构。 我需要使用C＃阅读它们，速度是一个问题。

我已经在C＃中完成了以下操作

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Size = 108)]
public struct PreIndexStruct {
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 96)]
    public string Key;
    public long Offset;
    public int Count;
}

然后我使用以下命令从文件中读取数据

using (BinaryReader br = new BinaryReader(
       new FileStream(pathToFile, FileMode.Open, FileAccess.Read, 
                      FileShare.Read, bufferSize))) 
{
    long length = br.BaseStream.Length;
    long position = 0;

    byte[] buff = new byte[structSize];
    GCHandle buffHandle = GCHandle.Alloc(buff, GCHandleType.Pinned);
    while (position < length) {
        br.Read(buff, 0, structSize);
        PreIndexStruct pis = (PreIndexStruct)Marshal.PtrToStructure(
            buffHandle.AddrOfPinnedObject(), typeof(PreIndexStruct));
        structures.Add(pis);

        position += structSize;
    }
    buffHandle.Free();
}

这非常有效，我可以从文件中检索数据。

我已经读过，如果我不使用GCHandle.Alloc / Marshal.PtrToStructure，而是使用C ++ / CLI或C＃不安全代码，则可以加快处理速度。 我发现了一些示例，但它们仅引用没有固定大小的数组的结构。

我的问题是，对于我的特殊情况，是否有使用C ++ / CLI或C＃不安全代码的更快处理方式？

编辑

其他性能信息（我使用过ANTS Performance Profiler 7.4）：

我的CPU时间的66％用于调用Marshal.PtrToStructure。

关于I / O，105 ms中只有6个用于读取文件。

Answer 1

在这种情况下，您不需要显式使用P / Invoke，因为您不必在托管代码和本机代码之间来回传递该结构。 因此，您可以这样做。 它将避免这种无用的GC句柄分配，而仅分配所需的内容。

public struct PreIndexStruct {
    public string Key;
    public long Offset;
    public int Count;
}

while (...) {
    ...
    PreIndexStruct pis = new PreIndexStruct();
    pis.Key = Encoding.Default.GetString(reader.ReadBytes(96));
    pis.Offset = reader.ReadInt64();
    pis.Count = reader.ReadInt32();
    structures.Add(pis);
}

我不确定您会比这快得多。

Answer 2

您可能更正确地想要使用非托管代码，这就是我要做的：

1. 创建一个C ++ / CLI项目，并移植现有的c＃代码并在其中运行
2. 确定瓶颈在哪里（使用探查器）
3. 用直接的C ++重写代码的这一部分，从C ++ / CLI代码中调用它，并确保它可以工作，然后再次对其进行概要分析
4. 用“ #pragma unmanaged”将新代码括起来
5. 再次剖析

您可能会获得某种程度的速度提高，但这可能并非您所期望的。

Answer 3

可以很轻松地快速读取一些结构数组，但是由于此技术需要可蓝调的类型，所以唯一的方法是为Key创建固定的字节缓冲区，而不是使用字符串。

如果这样做，则必须使用不安全的代码，因此可能真的不值得。

但是，仅出于好奇，这就是您可以对这些结构进行超duper快速读取和写入的方法，但以必须允许不安全的代码和大量的麻烦为代价：

using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Diagnostics.CodeAnalysis;
using System.IO;
using System.Runtime.InteropServices;


namespace Demo
{
    public static class Program
    {
        [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Size = 108)]
        public struct PreIndexStruct
        {
            public unsafe fixed byte Key[96];
            public long Offset;
            public int Count;
        }

        private static void Main(string[] args)
        {
            PreIndexStruct[] a = new PreIndexStruct[100];

            for (int i = 0; i < a.Length; ++i)
            {
                a[i].Count = i;

                unsafe
                {
                    fixed (byte* key = a[i].Key)
                    {
                        for (int j = 0; j < 10; ++j)
                        {
                            key[j] = (byte)i;
                        }
                    }
                }
            }

            using (var output = File.Create(@"C:\TEST\TEST.BIN"))
            {
                FastWrite(output, a, 0, a.Length);
            }

            using (var input = File.OpenRead(@"C:\TEST\TEST.BIN"))
            {
                var b = FastRead<PreIndexStruct>(input, a.Length);

                for (int i = 0; i < b.Length; ++i)
                {
                    Console.Write("Count = " + b[i].Count + ", Key =");

                    unsafe
                    {
                        fixed (byte* key = b[i].Key)
                        {
                            // Here you would access the bytes in Key[], which would presumably be ANSI chars.

                            for (int j = 0; j < 10; ++j)
                            {
                                Console.Write(" " + key[j]);
                            }
                        }
                    }

                    Console.WriteLine();
                }
            }
        }

        /// <summary>
        /// Writes a part of an array to a file stream as quickly as possible,
        /// without making any additional copies of the data.
        /// </summary>
        /// <typeparam name="T">The type of the array elements.</typeparam>
        /// <param name="fs">The file stream to which to write.</param>
        /// <param name="array">The array containing the data to write.</param>
        /// <param name="offset">The offset of the start of the data in the array to write.</param>
        /// <param name="count">The number of array elements to write.</param>
        /// <exception cref="IOException">Thrown on error. See inner exception for <see cref="Win32Exception"/></exception>

        [SuppressMessage("Microsoft.Reliability", "CA2001:AvoidCallingProblematicMethods", MessageId="System.Runtime.InteropServices.SafeHandle.DangerousGetHandle")]

        public static void FastWrite<T>(FileStream fs, T[] array, int offset, int count) where T: struct
        {
            int sizeOfT = Marshal.SizeOf(typeof(T));
            GCHandle gcHandle = GCHandle.Alloc(array, GCHandleType.Pinned);

            try
            {
                uint bytesWritten;
                uint bytesToWrite = (uint)(count * sizeOfT);

                if
                (
                    !WriteFile
                    (
                        fs.SafeFileHandle.DangerousGetHandle(),
                        new IntPtr(gcHandle.AddrOfPinnedObject().ToInt64() + (offset*sizeOfT)),
                        bytesToWrite,
                        out bytesWritten,
                        IntPtr.Zero
                    )
                )
                {
                    throw new IOException("Unable to write file.", new Win32Exception(Marshal.GetLastWin32Error()));
                }

                Debug.Assert(bytesWritten == bytesToWrite);
            }

            finally
            {
                gcHandle.Free();
            }
        }

        /// <summary>
        /// Reads array data from a file stream as quickly as possible,
        /// without making any additional copies of the data.
        /// </summary>
        /// <typeparam name="T">The type of the array elements.</typeparam>
        /// <param name="fs">The file stream from which to read.</param>
        /// <param name="count">The number of elements to read.</param>
        /// <returns>
        /// The array of elements that was read. This may be less than the number that was
        /// requested if the end of the file was reached. It may even be empty.
        /// NOTE: There may still be data left in the file, even if not all the requested
        /// elements were returned - this happens if the number of bytes remaining in the
        /// file is less than the size of the array elements.
        /// </returns>
        /// <exception cref="IOException">Thrown on error. See inner exception for <see cref="Win32Exception"/></exception>

        [SuppressMessage("Microsoft.Reliability", "CA2001:AvoidCallingProblematicMethods", MessageId="System.Runtime.InteropServices.SafeHandle.DangerousGetHandle")]

        public static T[] FastRead<T>(FileStream fs, int count) where T: struct
        {
            int sizeOfT = Marshal.SizeOf(typeof(T));

            long bytesRemaining  = fs.Length - fs.Position;
            long wantedBytes     = count * sizeOfT;
            long bytesAvailable  = Math.Min(bytesRemaining, wantedBytes);
            long availableValues = bytesAvailable / sizeOfT;
            long bytesToRead     = (availableValues * sizeOfT);

            if ((bytesRemaining < wantedBytes) && ((bytesRemaining - bytesToRead) > 0))
            {
                Debug.WriteLine("Requested data exceeds available data and partial data remains in the file.", "Dmr.Common.IO.Arrays.FastRead(fs,count)");
            }

            T[] result = new T[availableValues];

            GCHandle gcHandle = GCHandle.Alloc(result, GCHandleType.Pinned);

            try
            {
                uint bytesRead = 0;

                if
                (
                    !ReadFile
                    (
                        fs.SafeFileHandle.DangerousGetHandle(),
                        gcHandle.AddrOfPinnedObject(),
                        (uint)bytesToRead,
                        out bytesRead,
                        IntPtr.Zero
                    )
                )
                {
                    throw new IOException("Unable to read file.", new Win32Exception(Marshal.GetLastWin32Error()));
                }

                Debug.Assert(bytesRead == bytesToRead);
            }

            finally
            {
                gcHandle.Free();
            }

            return result;
        }


        /// <summary>See the Windows API documentation for details.</summary>

        [SuppressMessage("Microsoft.Interoperability", "CA1415:DeclarePInvokesCorrectly")]
        [DllImport("kernel32.dll", SetLastError=true)]
        [return: MarshalAs(UnmanagedType.Bool)]

        private static extern bool ReadFile
        (
            IntPtr hFile,
            IntPtr lpBuffer,
            uint nNumberOfBytesToRead,
            out uint lpNumberOfBytesRead,
            IntPtr lpOverlapped
        );

        /// <summary>See the Windows API documentation for details.</summary>

        [SuppressMessage("Microsoft.Interoperability", "CA1415:DeclarePInvokesCorrectly")]
        [DllImport("kernel32.dll", SetLastError=true)]
        [return: MarshalAs(UnmanagedType.Bool)]

        private static extern bool WriteFile
        (
            IntPtr hFile,
            IntPtr lpBuffer,
            uint nNumberOfBytesToWrite,
            out uint lpNumberOfBytesWritten,
            IntPtr lpOverlapped
        );
    }
}