读取包含单词“ upper”的.dat文件会导致问题

Question

This is the code I use to read in .dat files:

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('data.dat', sep='\s+')

plt.figure()
plt.plot(df['yh_center[2]'], df['tot_scale01[4]'], 'ro')
plt.xlabel('yh_center')
plt.ylabel('tot_scale01[4]')
plt.savefig('name.pdf')
plt.show()

It reads in correctly when there is no lonely word "upper" in the dat. file. However I will have many .dat files that have the word "upper" involved that cause errors.

The .dat file looks like:

#labels: yh_lower[1]   yh_center[2]   yh_upper[3]   tot_scale01[4]
#neval: 200000
#overflow:lower center upper    0.0000000000E+000    0.0000000000E+000    0.0000000000E+000    0.0000000000E+000

then on line 4 the actual data comes in

-4.4000000000E+000   -4.3000000000E+000   -4.2000000000E+000    0.0000000000E+000

Then i have 43 lines more of the data and the final line is

#nx: 3

(where actually there are over 100 columns of data, but that should change...the principle should be shown with the first 4 columns)

The full error report is

Traceback (most recent call last):

  File "<ipython-input-1-00512d6cb966>", line 1, in <module>
    runfile('Private, so deleted that one here')

  File "/usr/lib/python3.4/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 685, in runfile
    execfile(filename, namespace)

  File "/usr/lib/python3.4/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 85, in execfile
    exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)

  File "/home/ttp/manbrunn/Documents/NNLOJETexe/HJtest/funktioniert.py", line 15, in <module>
    plt.plot(df['yh_center[2]'], df['tot_scale01[4]'], 'ro')

  File "/usr/lib64/python3.4/site-packages/matplotlib/pyplot.py", line 3099, in plot
    ret = ax.plot(*args, **kwargs)

  File "/usr/lib64/python3.4/site-packages/matplotlib/axes/_axes.py", line 1374, in plot
    self.add_line(line)

  File "/usr/lib64/python3.4/site-packages/matplotlib/axes/_base.py", line 1504, in add_line
    self._update_line_limits(line)

  File "/usr/lib64/python3.4/site-packages/matplotlib/axes/_base.py", line 1515, in _update_line_limits
    path = line.get_path()

  File "/usr/lib64/python3.4/site-packages/matplotlib/lines.py", line 874, in get_path
    self.recache()

  File "/usr/lib64/python3.4/site-packages/matplotlib/lines.py", line 575, in recache
    x = np.asarray(xconv, np.float_)

  File "/usr/lib64/python3.4/site-packages/numpy/core/numeric.py", line 462, in asarray
    return array(a, dtype, copy=False, order=order)

ValueError: could not convert string to float: 'upper'

Answer 1

You could pre-parse the data by removing any leading text from each line before passing it to pandas, for example:

import pandas as pd
import re

with open('data.dat') as f_input:
    header = next(f_input).split()[1:]
    next(f_input)
    next(f_input)

    data = [line.strip().split() for line in f_input]

df = pd.DataFrame(data[:-1], dtype='float', columns=header)

This version is able to also extract the column names from the first row.

---

If the upper only appears on the third line (and not randomly throughout the file), you could simply use skiprows to tell Pandas to start reading from the forth line onwards, and as you have a #nx: 3 in the final line, use skipfooter to skip that too:

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('data.dat', sep='\s+', skiprows=3, skipfooter=1, header=None)

Tested on Pandas 0.22.0