使用python搜索极大的文本文件

Question

I have a large 40 million line, 3 gigabyte text file (probably wont be able to fit in memory) in the following format:

399.4540176 {Some other data}
404.498759292 {Some other data}
408.362737492 {Some other data}
412.832976111 {Some other data}
415.70665675 {Some other data}
419.586515381 {Some other data}
427.316825959 {Some other data}
.......

Each line starts off with a number and is followed by some other data. The numbers are in sorted order. I need to be able to:

1. Given a number x and and a range y , find all the lines whose number is within y range of x . For example if x=20 and y=5 , I need to find all lines whose number is between 15 and 25 .
2. Store these lines into another separate file.

What would be an efficient method to do this without having to trawl through the entire file?

Answer 1

If you don't want to generate a database ahead of time for line lengths, you can try this:

import os
import sys

# Configuration, change these to suit your needs
maxRowOffset = 100  #increase this if some lines are being missed
fileName = 'longFile.txt'
x = 2000
y = 25

#seek to first character c before the current position
def seekTo(f,c):
    while f.read(1) != c:
        f.seek(-2,1)

def parseRow(row):
    return (int(row.split(None,1)[0]),row)

minRow = x - y
maxRow = x + y
step = os.path.getsize(fileName)/2.
with open(fileName,'r') as f:
    while True:
        f.seek(int(step),1)
        seekTo(f,'\n')
        row = parseRow(f.readline())
        if row[0] < minRow:
            if minRow - row[0] < maxRowOffset:
                with open('outputFile.txt','w') as fo:
                    for row in f:
                        row = parseRow(row)
                        if row[0] > maxRow:
                            sys.exit()
                        if row[0] >= minRow:
                            fo.write(row[1])
            else:
                step /= 2.
                step = step * -1 if step < 0 else step
        else:
            step /= 2.
            step = step * -1 if step > 0 else step

It starts by performing a binary search on the file until it is near (less than maxRowOffset ) the row to find. Then it starts reading every line until it finds one that is greater than xy . That line, and every line after it are written to an output file until a line is found that is greater than x+y , and which point the program exits.

I tested this on a 1,000,000 line file and it runs in 0.05 seconds. Compare this to reading every line which took 3.8 seconds.

Answer 2

You need random access to the lines which you won't get with a text files unless the lines are all padded to the same length.

One solution is to dump the table into a database (such as SQLite) with two columns, one for the number and one for all the other data (assuming that the data is guaranteed to fit into whatever the maximum number of characters allowed in a single column in your database is). Then index the number column and you're good to go.

Without a database, you could read through file one time and create an in-memory data structure with pairs of values showing containing (number, line-offset). You calculate the line-offset by adding the lengths of each row (including line end). Now you can binary search these value pairs on number and randomly access the lines in the file using the offset. If you need to repeat the search later, pickle the in-memory structure and reload for later re-use.

This reads the entire file (which you said you don't want to do), but does so only once to build the index. After that you can execute as many requests against the file as you want and they will be very fast.

Note that this second solution is essentially creating a database index on your text file.

Rough code to create the index in second solution:

 import Pickle

 line_end_length = len('\n') # must be a better way to do this!
 offset = 0
 index = [] # probably a better structure to use than a list

 f = open(filename)
 for row in f:
     nbr = float(row.split(' ')[0])
     index.append([nbr, offset])
     offset += len(row) + line_end_length

 Pickle.dump(index, open('filename.idx', 'wb')) # saves it for future use

Now, you can perform a binary search on the list. There's probably a much better data structure to use for accruing the index values than a list, but I'd have to read up on the various collection types.

Answer 3

Since you want to match the first field, you can use gawk :

$ gawk '{if ($1 >= 15 && $1 <= 25) { print }; if ($1 > 25) { exit }}' your_file

Edit: Taking a file with 261,775,557 lines that is 2.5 GiB big, searching for lines 50,010,015 to 50,010,025 this takes 27 seconds on my Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz . Sounds good enough for me.

Answer 4

In order to find the line that starts with the number just above your lower limit, you have to go through the file line by line until you find that line. No other way, ie all data in the file has to be read and parsed for newline characters.

We have to run this search up to the first line that exceeds your upper limit and stop. Hence, it helps that the file is already sorted. This code will hopefully help:

with open(outpath) as outfile:
    with open(inpath) as infile:
        for line in infile:
            t = float(line.split()[0])
            if lower_limit <= t <= upper_limit:
                outfile.write(line)
            elif t > upper_limit:
                break

I think theoretically there is no other option.