将大的二进制文件（5GB）读入C＃的字节数组中？

Question

I have a recording file (Binary file) more than 5 GB, i have to read that file and filter out the data needed to be send to server.

Problem is byte[] array supports till 2GB of file data . so just need help if someone had already dealt with this type of situation.

using (FileStream str = File.OpenRead(textBox2.Text))
{
       int itemSectionStart = 0x00000000;
       BinaryReader breader = new BinaryReader(str);
       breader.BaseStream.Position = itemSectionStart;
       int length = (int)breader.BaseStream.Length;
       byte[] itemSection = breader.ReadBytes(length );  //first frame data
}

issues:

1: Length is crossing the range of integer.
2: tried using long and unint but byte[] only supports integer

Edit.

Another approach i want to give try, Read data on frame buffer basis, suppose my frame buffer size is 24000 . so byte array store that many frames data and then process the frame data and then flush out the byte array and store another 24000 frame data. till keep on going till end of binary file..

Answer 1

As said in comments, I think you have to read your file with a stream. Here is how you can do this:

int nbRead = 0;
var step = 10000;
byte[] buffer = new byte[step];
do
{
    nbRead = breader.Read(buffer, 0, step);
    hugeArray.Add(buffer);

    foreach(var oneByte in hugeArray.SelectMany(part => part))
    {
        // Here you can read byte by byte this subpart
    }
}
while (nbRead > 0);

If I well understand your needs, you are looking for a specific pattern into your file?

I think you can do it by looking for the start of your pattern byte by byte. Once you find it, you can start reading the important bytes. If the whole important data is greater than 2GB, as said in the comments, you will have to send it to your server in several parts.

Answer 2

See you can not read that much big file at once, so you have to either split the file in small portions and then process the file.

 OR

Read file using buffer concept and once you are done with that buffer data then flush out that buffer.

I faced the same issue, so i tried the buffer based approach and it worked for me.

         FileStream inputTempFile = new FileStream(Path, FileMode.OpenOrCreate, FileAccess.Read);
           Buffer_value = 1024;
            byte[] Array_buffer = new byte[Buffer_value];
            while ((bytesRead = inputTempFile.Read(Array_buffer, 0, Buffer_value)) > 0)
            {
               for (int z = 0; z < Array_buffer.Length; z = z + 4)
               {
                  string temp_id = BitConverter.ToString(Array_buffer, z, 4);
                  string[] temp_strArrayID = temp_id.Split(new char[] { '-' });
                  string temp_ArraydataID = temp_strArrayID[0] + temp_strArrayID[1] + temp_strArrayID[2] + temp_strArrayID[3];
               }
            }

this way you can process your data.

For my case i was trying to store buffer read data in to a List, it will work fine till 2GB data after that it will throw memory exception.

The approach i followed, read the data from buffer and apply needed filters and write filter data in to a text file and then process that file.

//text file approach

           FileStream inputTempFile = new FileStream(Path, FileMode.OpenOrCreate, FileAccess.Read);
           Buffer_value = 1024;
            StreamWriter writer = new StreamWriter(Path, true);
            byte[] Array_buffer = new byte[Buffer_value];
            while ((bytesRead = inputTempFile.Read(Array_buffer, 0, Buffer_value)) > 0)
            {
               for (int z = 0; z < Array_buffer.Length; z = z + 4)
               {
                 string temp_id = BitConverter.ToString(Array_buffer, z, 4);
                 string[] temp_strArrayID = temp_id.Split(new char[] { '-' });
                 string temp_ArraydataID = temp_strArrayID[0] + temp_strArrayID[1] + temp_strArrayID[2] + temp_strArrayID[3];
                 if(temp_ArraydataID =="XYZ Condition")
                 { 
                     writer.WriteLine(temp_ArraydataID);
                 }
               }

            }
           writer.Close();