在读取 csv 时停止 pandas 对数据进行分组

Question

I have a problem, when I read a csv file with pandas and assign it the header to be row 0 with the following:

df = pd.read_csv(fileName, header = [0])

The first x columns on each row is being grouped and surrounded by parenthesis. For example, if I have the following:

Version, temp, altitude, oxygen, pressure, gas_temp, NH3, NO2
1,189,2980,489.9,594,345,345,22,00
2,11,33,423,554.9,2345,32,22,01

When I try to print each row from the data frame, I get this:

(1,189,2980,489.9,594) 345 345 22 00
(2,11,33,423,554.9) 2345 32 22 01

And if I call df['temp'] I get back [345, 2345] which is not correct because pandas is grouping the first x columns together.

Answer 1

The firsr important discrepancy in your data sample is that it:

• contains only 8 column names,
• but there are 9 data columns.

Another disorder is that column names should be separated only with commas, but your input contains also spaces.

The cumulated effect of the above is that read_csv is "fooled" and can read data in such a weird way. Actually, at the first attempt I replicated your case, but the next time (and all following times) I got an apparently proper result.

The reason I wrote "apparently" is that when you print df.columns , you wil see another flaw:

Index(['Version', ' temp', ' altitude', ' oxygen', ' pressure', ' gas_temp',
       ' NH3', ' NO2'],
      dtype='object')

ie column names contain initial spaces , so an attempt to refer eg to temp column throws exception:

AttributeError: 'DataFrame' object has no attribute 'temp'

One thing you can do to read this file correctly is to pass skipinitialspace=True parameter:

df = pd.read_csv(fileName, skipinitialspace=True, header=[0])

The result of reading is:

   Version  temp  altitude  oxygen  pressure  gas_temp  NH3  NO2
1      189  2980     489.9   594.0       345       345   22    0
2       11    33     423.0   554.9      2345        32   22    1

When you print df.columns , you will see that column names are without initial spaces, so now these extra spaces in the header row are stripped.

Another detail in the way how read_csv operates is that it matches column names with data columns from the end , so the "additional" column (in data rows) is taken as the index column, with no name.

You can also add index_col=[0] parameter:

df = pd.read_csv(filename, skipinitialspace=True, index_col=[0], header=[0])

to specify explicitely that the initial column is the index.