使用python提高读写txt文件的性能

Question

I am using python to manage txt files, read and write several times. But if I have 1000 txt files my code is taking so long time to manage the files.

How can I improve the performance to manage with these files?

I have files with this information:

position   temperature
0,0        30,10
1,0        45,12
2,0        20,45 (...)

In the first place I need to remove the lines with strings. For taht I search the strings and create new txt file and copy the information without the lines with strings to that new txt file. I get this:

0,0       30,10
1,0       45,12
2,0       20,45 (...)

Then I replace the , by . in all the files, again creating a new file and coping the information with points to these new files. I get this:

0.0      30.10
1.0      45.12
2.0      20.45 (...)

Then I need to restrict the information with a minimal position value (a) and a maximum value (b). So in the first column I just want the lines between a and b. Again I create new files and copy the information that I want to these files.

Finally I add a new column to each file with some information.

So, I think that the time consuming in my code is related to the times that I create new files, copy the information and replace the old ones with these new ones.

Maybe this all can be done in one step. I just need to know if it is possible.

Thanks

Answer 1

You no need to create new files just to delete first lines or replace commas to dots in lines. You could do all work in memory, ie read data from file, replace commas to dots, convert values to float, sort them, trim min and max values and write result to file, like this:

input_file = open('input_file', 'r')
data = []
input_file.readline() # first line with titles
for line in input_file: # lines with data
    data.append(map(lambda x: float(x.replace(',', '.'), line.split()))
input_file.close()

data.sort(key=lambda x: x[1])
data = data[1:-1]

result_file = open('result_file', 'w')
result_file.writelines(['\t'.join(row) for row in data])

result_file.close()

Answer 2

You're definitely doing it the hard way... There are a lot of detail informations missing from your problem's description, but assuming a few things - like the headers are always on the first line, "position" and "temperature" are supposed to be floats etc - here's a code example doing mostly what you describe in a single pass:

import sys
from itertools import ifilter

def parse(path):
    with open(path) as f:
        # skip the headers
        f.next()

        # parse data
        for lineno, line in enumerate(f, 1):
            try:
                position, temperature = line.split()
                position = float(position.replace(",", "."))
                temperature = float(temperature.replace(",","."))
            except ValueError as e:
                raise ValueError("Invalid line at %s:#%s" % (path, lineno))

            yield position, temperature


def process(inpath, outpath, minpos, maxpos):
    # warning: exact comparisons on floating points numbers
    # are not safe
    filterpos = lambda r: minpos <= r[0] <= maxpos 

    with open(outpath, "w") as outfile:
        for pos, temp in ifilter(filterpos, parse(inpath)):
            extra_col = compute_something_from(pos, temp)
            out.writeline("{}\t{}\t{}\t{}\n".format(pos, temp, extra_col))


def compute_something_from(pos, temp):
    # anything
    return pos * (temp / 3.)


def main(*args):
    # TODO : 
    # - clean options / args handling
    # - outfile naming ? 
    minpos = float(args[0])
    maxpos = float(args[1])
    for inpath in args[2:]:
        outpath = inpath + ".tmp"
        process(inpath, outpath, minpos, maxpos)

if __name__ == "__main__":
    main(*sys.argv[1:])