在Python中读取以空格分隔的数据的最快方法

Question

I have some data that is separated by white space from which I want to extract certain columns. In the past I have always used something like the following in Python, in which I have removed the non-essentials:

for line in open(f,'r'):
    l = line.split()
    print " ".join(l[1:3])

I'm wondering though whether this is the fastest way to do this. If I compare to another software package (written in C) that reads the same data, my code is significantly slower. Is this simply because I/O in C is faster or am I writing suboptimal code?

Answer 1

You can get the expected columns in a list using list comprehensions.

expectedColumns = [" ".join(x) for x in [line.split()[0:2] for line in file("testFile",'r').readlines()]]

If you want to print the columns inside LC, you can do this :)

from __future__ import print_function
[print(" ".join(x)) for x in [line.split()[0:2] for line in file("testFile",'r').readlines()]]

Answer 2

When you directly iterate over a file , the file is read line by line. This helps for huge files but at the penalty of IO, even though if it implements read ahead buffer. AFAIK, internally it uses seek and tell when iterating.

If you do read() , it will read the entire content once but at the cost of memory. In your case you can do read().split('\\n') or readlines() (preferred) and it will be faster than iterating over the file directly.

In addition to the above, please use context managers when dealing with files, so they are closed once done.

Docs

Answer 3

You might want to look at the CSV module. csv.reader is implemented in c and should be faster than using pure python.

import csv
with open(f, 'rb') as file:
    r = csv.reader(file, delimiter=' ')
    for line in r:
        print ' '.join(line[1:3])