Skip `# ` character when reading header with pandas read_csv

Question

I have a file that looks like this:

# Time                  Cm                      Cd                      Cl                      Cl(f)                   Cl(r)                   Cm      Cd      Cl      Cl(f)   Cl(r)
1.000000000000e+01      -5.743573465913e-01     -5.860160539688e-01     -1.339511756657e+00     -1.244113224920e+00     -9.539853173733e-02
2.000000000000e+01      6.491397073110e-02      1.320098727949e-02      6.147195262817e-01      3.722737338720e-01      2.424457924098e-01
3.000000000000e+01      3.554043329234e-02      4.296597501519e-01      7.901295853361e-01      4.306052259604e-01      3.595243593757e-01

Is there any way I can tell pandas that Time is the first column name?

I read it this way

dat = pd.read_csv('%sdt.dat'%s, delim_whitespace=True)

Which is somehow telling pandas that the first column is named # :

dat.columns
Index(['#', 'Time', 'Cm', 'Cd', 'Cl', 'Cl(f)', 'Cl(r)', 'Cm.1', 'Cd.1', 'Cl.1', 'Cl(f).1', 'Cl(r).1'],
      dtype='object')

How can I tell pandas ' read_csv to ignore the first two characters in the header or otherwise get the column names I want from read_csv ?

Answer 1

Here is one potential work-around:

headers = pd.read_csv('%sdt.dat'%s, delim_whitespace=True, nrows=0).columns[1:]
dat = pd.read_csv('%sdt.dat'%s, delim_whitespace=True, header=None, skiprows=1, names=headers)

alternatively, you could fix the columns with some post-processing:

col_mapper = {old:new for old, new in zip(dat.columns, dat.columns[1:])}
dat = dat.iloc[:, :-1].rename(col_mapper, axis=1)

Answer 2

Instead of using any whitespace as a separator you can specify that there must be at least 2 whitespace characters since your data appears to be separated by multiple spaces. This will name the first column '# Time' and you can rename it afterwards to remove the '# ' prefix:

df = pd.read_csv('%sdt.dat'%s, sep='\s{2,}', engine='python')
print(df)

   # Time        Cm        Cd        Cl     Cl(f)     Cl(r)  Cm.1  Cd.1  Cl.1  Cl(f).1  Cl(r).1
0    10.0 -0.574357 -0.586016 -1.339512 -1.244113 -0.095399   NaN   NaN   NaN      NaN      NaN
1    20.0  0.064914  0.013201  0.614720  0.372274  0.242446   NaN   NaN   NaN      NaN      NaN
2    30.0  0.035540  0.429660  0.790130  0.430605  0.359524   NaN   NaN   NaN      NaN      NaN

df.columns = ['Time'] + list(df.columns[1:])
print(df)

   Time        Cm        Cd        Cl     Cl(f)     Cl(r)  Cm.1  Cd.1  Cl.1  Cl(f).1  Cl(r).1
0  10.0 -0.574357 -0.586016 -1.339512 -1.244113 -0.095399   NaN   NaN   NaN      NaN      NaN
1  20.0  0.064914  0.013201  0.614720  0.372274  0.242446   NaN   NaN   NaN      NaN      NaN
2  30.0  0.035540  0.429660  0.790130  0.430605  0.359524   NaN   NaN   NaN      NaN      NaN