二维ndarray的Numpy fromfile的填充顺序是什么？

Question

I'm trying to read a structured binary file using the numpy.fromfile() function. In my case, I have a numpy.dtype() which is used to define a user defined data type to use with np.fromfile() .
I will reproduce the relevant part of the data structure here( for the full structure is rather long):

('RawData', np.int32, (2, BlockSize))  

this will read BlockSize*2 number of int32s into the field RawData , will produce a 2xBlockSize matrix. This is where I am having trouble because I want to replicate the behavior of Matlab's fread() function, in which the matric is filled in column order . As for NumPy's fromfile() , this isn't mentioned (at least I couldn't find it).

It doesn't matter NumPy's fromfile() should work like Matlab's fread() , but I have to know how NumPy's fromfile() works to code accordingly.

Now, the question is, what is the fill order of a 2-D array in the NumPy fromfile() function when using a custom data type?

Answer 1

fromfile and tofile read/write flat, 1d, arrays:

In [204]: x = np.arange(1,11).astype('int32')                                                          
In [205]: x.tofile('data615')                                                                          

fromfile returns a 1d array:

In [206]: np.fromfile('data615',np.int32)                                                              
Out[206]: array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10], dtype=int32)

x.reshape(2,5).tofile(...) would save the same thing. tofile does not save dtype or shape information.

reshaped to 2d, the default order is 'C':

In [207]: np.fromfile('data615',np.int32).reshape(2,5)                                                 
Out[207]: 
array([[ 1,  2,  3,  4,  5],
       [ 6,  7,  8,  9, 10]], dtype=int32)

but it can be changed to MATLAB like:

In [208]: np.fromfile('data615',np.int32).reshape(2,5, order='F')                                      
Out[208]: 
array([[ 1,  3,  5,  7,  9],
       [ 2,  4,  6,  8, 10]], dtype=int32)

The underlying databuffer is the same, just a 1d array of bytes.

edit

The file could be read as a 2 integer structure:

In [249]: np.fromfile('data615','i4,i4')                                                               
Out[249]: 
array([(1,  2), (3,  4), (5,  6), (7,  8), (9, 10)],
      dtype=[('f0', '<i4'), ('f1', '<i4')])
In [250]: _['f0']                                                                                      
Out[250]: array([1, 3, 5, 7, 9], dtype=int32)

It's still a 1d array, but with numbers grouped by 2s.

Converting to complex:

In [252]: xx = np.fromfile('data615','i4,i4')                                                          
In [253]: xx['f0']+1j*xx['f1']                                                                         
Out[253]: array([1. +2.j, 3. +4.j, 5. +6.j, 7. +8.j, 9.+10.j])
In [254]: _.dtype                                                                                      
Out[254]: dtype('complex128')

If the data had been saved as floats, we could load them as complex directly:

In [255]: x.astype(np.float32).tofile('data615f')                                                      
In [257]: xx = np.fromfile('data615f',np.complex64)                                                    
In [258]: xx                                                                                           
Out[258]: array([1. +2.j, 3. +4.j, 5. +6.j, 7. +8.j, 9.+10.j], dtype=complex64)

Another way to get the complex from the integer sequence:

In [261]: np.fromfile('data615', np.int32).reshape(5,2)                                                
Out[261]: 
array([[ 1,  2],
       [ 3,  4],
       [ 5,  6],
       [ 7,  8],
       [ 9, 10]], dtype=int32)
In [262]: xx = np.fromfile('data615', np.int32).reshape(5,2)                                           
In [263]: xx[:,0]+1j*xx[:,1]                                                                           
Out[263]: array([1. +2.j, 3. +4.j, 5. +6.j, 7. +8.j, 9.+10.j])

Answer 2

By default, when creating a new 2-d array, NumPy will use "C" ordering, which is row-major . That is the opposite of the order used by Matlab.

For example, if BlockSize is 4, and the raw data is

0 1 2 3 4 5 6 7

then the 2 x 4 array will be

[[0, 1, 2, 3],
 [4, 5, 6, 7]]

With Matlab and that same raw data, the 2 x 4 array would be

[[0, 2, 4, 6],
 [1, 3, 5, 7]]