在python中读取和保存具有可变列数的数据文件

Question

I have a space separated data file that looks like this (just a slice)

Wavelength  Ele   Excit   loggf       D0        
11140.324   108.0 3.44     -7.945    4.395
11140.357    26.1 12.09    -2.247
11140.361   108.0 2.39     -8.119    4.395
11140.365    25.0 5.85    -9.734
11140.388    23.0 4.56    -4.573
11140.424   608.0 5.12    -10.419    11.09 
11140.452   606.0 2.12    -11.054     6.25 
11140.496   108.0 2.39    -8.119      4.395
11140.509   606.0 1.70    -7.824      6.25 

Part 1

First I would like to read the file a lá np.loadtxt . This does not work, so I tried with

d = np.genfromtxt('file.dat', skiprows=1, filling_value=0.0, missing_values=' ')

and different versions of that. All gave errors: Line #3 (got 4 columns instead of 5) . I think I'm close to be able to read the file. Note, that I prefer a solution with something like np.genfromtxt rather than openning the file and go through it line by line:

with open('test.dat', 'r') as lines:
    for line in lines:
        # put numbers in arrays/lists

Part 2

After reading the file succesfully, I need to save it in a specific format. Very briefly, this file will be a input for a Fotran program, with 10 spaces per column for numbers. Without the last column ( D0 ), I can use (there is a column I don't use, hence the '%27.1f' )

fmt_ = ('%9.2f', '%7.1f', '%11.2f','%10.3f', '%27.1f')
np.savetxt('output.dat', data, fmt=fmt_)

But I suspect this wouldn't work either. So s np.genfromtxt for saving could be helpful.

Help for it all, one part or just some guidance are appreciated.

Answer 1

Part 1:

Use pandas. It is designed specifically to handle this sort of scenario:

import pandas as pd
df = pd.read_csv('test.csv', sep='\s+')
print(df)

gives you:

   Wavelength    Ele  Excit   loggf      D0
0   11140.324  108.0   3.44  -7.945   4.395
1   11140.357   26.1  12.09  -2.247     NaN
2   11140.361  108.0   2.39  -8.119   4.395
3   11140.365   25.0   5.85  -9.734     NaN
4   11140.388   23.0   4.56  -4.573     NaN
5   11140.424  608.0   5.12 -10.419  11.090
6   11140.452  606.0   2.12 -11.054   6.250
7   11140.496  108.0   2.39  -8.119   4.395
8   11140.509  606.0   1.70  -7.824   6.250

Part 2

You can also use pandas for this, although it is a bit more complex to get the formatting correct:

formatters  = ['{: >9.2f}'.format, '{: >7.1f}'.format, 
               '{: >11.2f}'.format,'{: >10.3f}'.format, 
               lambda x: ' '*27 if np.isnan(x) else '{: >27.1f}'.format(x)]

lines = df.to_string(index=False, header=False, formatters=formatters)

with open('out.dat', 'w') as outfile:
    outfile.write(lines)

Gives you:

 11140.32   108.0        3.44     -7.945                         4.4
 11140.36    26.1       12.09     -2.247                            
 11140.36   108.0        2.39     -8.119                         4.4
 11140.36    25.0        5.85     -9.734                            
 11140.39    23.0        4.56     -4.573                            
 11140.42   608.0        5.12    -10.419                        11.1
 11140.45   606.0        2.12    -11.054                         6.2
 11140.50   108.0        2.39     -8.119                         4.4
 11140.51   606.0        1.70     -7.824                         6.2

Answer 2

Here's part of a sample run with your data.

In [62]: txt=b"""Wavelength  Ele   Excit   loggf       D0        
11140.324   108.0 3.44     -7.945    4.395
...
11140.509   606.0 1.70    -7.824      6.25 """

In [63]: txt=txt.splitlines()

In [64]: def foo(astr):
    # add a 'NaN' field to the short lines
    if len(astr)<35:
        astr += b'  NaN'  # or filler of your choice
    return astr
   ....: 

In [65]: data=np.loadtxt([foo(t) for t in txt], skiprows=1)

In [66]: data
Out[66]: 
array([[  1.11403240e+04,   1.08000000e+02,   3.44000000e+00,
         -7.94500000e+00,   4.39500000e+00],
       [  1.11403570e+04,   2.61000000e+01,   1.20900000e+01,
         -2.24700000e+00,              nan],
        ...
       [  1.11405090e+04,   6.06000000e+02,   1.70000000e+00,
         -7.82400000e+00,   6.25000000e+00]])

In [67]: np.savetxt('test.dat',x,fmt=fmt_)

In [69]: cat test.dat
 11140.32   108.0        3.44     -7.945                         4.4
 11140.36    26.1       12.09     -2.247                         nan
 11140.36   108.0        2.39     -8.119                         4.4
 11140.36    25.0        5.85     -9.734                         nan
 ...
 11140.51   606.0        1.70     -7.824                         6.2

The file can be passed through foo like this:

with open('test.dat') as f: 
     xx = np.loadtxt((foo(t) for t in f),skiprows=1)

savetxt essentially does a row by row write , so it isn't hard to write your own version. eg

In [120]: asbytes=np.lib.npyio.asbytes

In [121]: fmt__='%9.2f  %7.1f  %11.2f  %10.3f  %10.1f'

In [122]: with open('test.dat','wb') as f: 
     for row in x:
        f.write(asbytes(fmt__%tuple(row)+'\n'))
   .....:         

In [123]: cat test.dat
 11140.32    108.0         3.44      -7.945         4.4
 11140.36     26.1        12.09      -2.247         nan
 11140.36    108.0         2.39      -8.119         4.4
 11140.36     25.0         5.85      -9.734         nan
 ...
 11140.51    606.0         1.70      -7.824         6.2

With this it wouldn't be hard to test each row, and use a different format for rows with a nan .