使用C ++时从二进制文件读取int错误

Question

I have a complex structured binary file. I created a parser in python to read the binary file and convert to the correct values and save the data to csv so that the values can be analyzed. This works well but some of the files are extremely large (ie 20+ Gb) and take many many hours to parse. I am trying to speed this up by implementing the same process in c++.

Below is an excerpt that reads a control word at the beginning of each logical record and specifies the size of the record. For a specific case the control word is 128 (a 4 byte, Big Endian, int). In python I do:

x = open(str(self.filename), "rb")
cw_d_type = np.dtype('>i4')
temp = np.frombuffer(x.read(cw_d_type.itemsize), dtype=cw_d_type)

The value in temp[0] after this is 128. Now when I attempt to do this same thing in c++ using the following code

#include <iostream>
#include <fstream>
#include <stdio.h>
#include <sstream>
#include <stdint.h>

using namespace std

struct control_word
{
    uint32_t chunk_size;
}

int main()
{
    // define my stream
    ifstream in_f("Y:/path_to_binary_file/binary_file", ios::binary | ios::in | ios::ate);

    // find the size of the file
    int file_size = in_f.tellg();

    // goto the beginning of the file
    in_f.seekg(0, std::ios::beg);

    in_f.read(reinterpret_cast<char*>(&cw), sizeof(cw));
    cout << cw.chunk_size << endl

    ... continue reading the rest of the structures

}

The result for cw.chunk_size = 2147483648. I know that I am reading the correct place in the file b/c the next structure that I read has a 32 bit string and it is being read properly, if I was not in the right location in the file then that result would not be correct.

If I change my control word structure from an int to a char[4] then the result is [0][0][0][-128] which is almost correct except the negative sign is there.

All of the doubles and floats that I read in are showing the same thing. The only thing that seems to be read properly is char values. It has been a number of years since I last programmed in c++. Is there something that I am forgetting to do to properly map my binary into my structures ??

I have read many questions concerning reading binary files and can't figure out why I am getting these weird values. The closest answer that I have found is here , and the solution was the user was not mapping the chunk of binary into the correct type. I know that this is not the case for me b/c in my python implementation I read the chunk as an int and get the value I am expecting.

Answer 1

According to documentation > for numpy.dtype specifies Big-Endian format. You are running your code most probably on Intel or compatible CPU which is Little-Endian. You need to convert your uint32_t field using ntohl() function:

in_f.read(reinterpret_cast<char*>(&cw), sizeof(cw));
cw.chunk_size = ntohl( cw.chunk_size );
cout << cw.chunk_size << endl;

Details about Endianness