什么时候应该使用numpy.genfromtxt而不是pandas.read_csv来读取csv文件？

Question

I was recently doing an image extraction part from a .csv file,the file contained a column named pixels with 48x48 values given as strings, so normally seeing a .csv file I used pandas.read_csv to try to convert pixels column to later on images, converting to PIL images.

import pandas as pd
data = pd.read_csv('fer2013.csv') # fer2013 competition dataset.
data.head()

        emotion pixels  Usage
    0   0   70 80 82 72 58 58 60 63 54 58 60 48 89 115 121...   Training
    1   0   151 150 147 155 148 133 111 140 170 174 182 15...   Training
    2   2   231 212 156 164 174 138 161 173 182 200 106 38...   Training
    3   4   24 32 36 30 32 23 19 20 30 41 21 22 32 34 21 1...   Training
    4   6   4 0 0 0 0 0 0 0 0 0 0 0 3 15 23 28 48 50 58 84...

But, I saw another guy use numpy.genfromtxt to load the csv file from the discussions:

data = np.genfromtxt('fer2013.csv',delimiter=',',dtype=None)

But, I don't understand what's the use of numpy.genfromtxt, I saw the examples on the scipy numpy.genfromtxt docs too,

I found the dtype naming methods to be great , but those are available in pd.read_csv too!

: np.genfromtxt

Would be great if someone could explain the need and use for numpy.genfromtxt load method, and where it would benefit on top of other methods for reading a file.

You can find the data here: fer2013 competition Kaggle

Answer 1

As I understand it , the pandas reader is a optimized program written in C and is faster in much situation. genfromtext is an old python fonction with less inferring skills, that you can forget if you have pandas.

In [45]: df=pd.DataFrame(np.arange(10**6).reshape(1000,1000))

In [46]: df.to_csv("data.csv")

In [47]: %time v=np.genfromtxt("data.csv",delimiter=',',dtype=int,skip_header=1)
Wall time: 5.62 s

In [48]: %time u=pd.read_csv("data.csv",engine='python')
Wall time: 3.97 s

In [49]: %time u=pd.read_csv("data.csv")
Wall time: 781 ms

The docs describe the engine option :

engine : {'c', 'python'}, optional

Parser engine to use. The C engine is faster while the python engine is currently more feature-complete.

Answer 2

I can't download the linked dataset, but tried to recreate it from your header:

In [2]: cat stack53997674.csv
emotion, pixels,  Usage
0,   "70 80 82 72 58 58 60 63 54 58 60 48 89 115 121",   Training
0,   "151 150 147 155 148 133 111 140 170 174 182 15",   Training
2,   "231 212 156 164 174 138 161 173 182 200 106 38",   Training
4,   "24 32 36 30 32 23 19 20 30 41 21 22 32 34 21 1",   Training
6,   "4 0 0 0 0 0 0 0 0 0 0 0 3 15 23 28 48 50 58 84",   Testing

With pandas:

In [11]: df = pd.read_csv("stack53997674.csv")
In [12]: df
Out[12]: 
   emotion     ...             Usage
0        0     ...          Training
1        0     ...          Training
2        2     ...          Training
3        4     ...          Training
4        6     ...           Testing

[5 rows x 3 columns]
In [13]: df.dtypes
Out[13]: 
emotion     int64
 pixels    object
  Usage    object
dtype: object

values is a 2d object dtype array, with strings in the 2nd column:

In [20]: df.values[:,1]
Out[20]: 
array(['   "70 80 82 72 58 58 60 63 54 58 60 48 89 115 121"',
       '   "151 150 147 155 148 133 111 140 170 174 182 15"',
       '   "231 212 156 164 174 138 161 173 182 200 106 38"',
       '   "24 32 36 30 32 23 19 20 30 41 21 22 32 34 21 1"',
       '   "4 0 0 0 0 0 0 0 0 0 0 0 3 15 23 28 48 50 58 84"'],
      dtype=object)

With genfromtxt :

In [21]: data = np.genfromtxt("stack53997674.csv", delimiter=',', names=True, dt
    ...: ype=None, encoding=None, autostrip=True)
In [22]: data
Out[22]: 
array([(0, '"70 80 82 72 58 58 60 63 54 58 60 48 89 115 121"', 'Training'),
       (0, '"151 150 147 155 148 133 111 140 170 174 182 15"', 'Training'),
       (2, '"231 212 156 164 174 138 161 173 182 200 106 38"', 'Training'),
       (4, '"24 32 36 30 32 23 19 20 30 41 21 22 32 34 21 1"', 'Training'),
       (6, '"4 0 0 0 0 0 0 0 0 0 0 0 3 15 23 28 48 50 58 84"', 'Testing')],
      dtype=[('emotion', '<i8'), ('pixels', '<U48'), ('Usage', '<U8')])
In [23]: data['pixels']
Out[23]: 
array(['"70 80 82 72 58 58 60 63 54 58 60 48 89 115 121"',
       '"151 150 147 155 148 133 111 140 170 174 182 15"',
       '"231 212 156 164 174 138 161 173 182 200 106 38"',
       '"24 32 36 30 32 23 19 20 30 41 21 22 32 34 21 1"',
       '"4 0 0 0 0 0 0 0 0 0 0 0 3 15 23 28 48 50 58 84"'], dtype='<U48')

pixels is a 1d array of string dtype. Both can be converted to/from the other dtype. And both will require similar processing to produce images.