使用Python读写固定格式（MODFLOW）文本文件

Question

I am trying to read, manipulate and write text files using python. These files contain numeric matrices and were generated from a FORTRAN groundwater flow code called MODFLOW, and have an unusual shape, because the matrix rows are split across several file lines so that there are no more than 7 values per line. So a matrix row with 37 columns is output as 5 lines of 7 values (fmt='%14.6E') followed by 1 line with 2 values. The next matrix row then starts on a new line.

I am trying to read two such files, each with 730 time steps x 49 rows x 37 columns (about 18 Mb). Then I want to multiply the data together elementwise and write the results into a new file with the same format.

I can do it line by line using csv.reader and then numpy.savetext but it is extremely slow. How can I do it with numpy (or similar) that will be faster? Thanks!

UPDATE:

I'm almost there, just need to get rid of the commas in my output file. Apparently this isn't currently possible with pandas, so I might have to do it aa separate operation.

SOLVED:

Obtain the pandas output as text and use replace() to get rid of the delimiters. Still fast.

import pandas as pd

root = 'Taupo'

rctrans = read_csv(root+'._rctrans', header=None, delim_whitespace=True)
rcmult = read_csv(root+'._rcmult', header=None, delim_whitespace=True)

# duplicate rcmult nsteps times to make it the same size as rctrans
nsteps = len(rctrans.index)/len(rcmult.index)    
rcmult = pd.concat([rcmult]*nsteps, ignore_index=True)

# multiply the arrays
rctrans = pd.DataFrame(rctrans.values*rcmult.values, columns=rctrans.columns, index=rctrans.index)

# write as csv with no delimiter
with open(root+'._rc','w') as w:
    w.write(rctrans.to_csv(header=False, index=False, float_format='%14.6E').replace(',',''))

Answer 1

I think any Python based file reader that handles the files line by line is going to have similar speed. Pandas supposedly has a faster CSV reader, but I'm not familiar with it. Do you have any sense of where your code is slow? reading the files? parsing? collecting values in a list/array?

For a start I'd try to write a reader that takes in 6 lines, splices them together to get the 37 numbers in one line. Then parse that and convert to a list of 37 floats. Finally append it to a master list.

Once I'm done 49 of those, create a (49,37) array, and save it or append it to another list that will hold all the time steps.

As noted in other SO questions about np.genfromtxt or np.loadtxt , they accept any iterator (or generator). So the input to the function could be this aggregator that turns 6 lines in to one line with 37 columns.

Without knowing more details about your current method, I can't say whether my suggestion is any faster. And without a similar test file, I really can't test alternatives. So at one level or other this is all speculative.