逐块读取文本文件-丢失数据？

Question

I have a huge text file that can be anywhere from 2 to 20 GBs. The content is a list of results from a given list of queries. I am trying to send it into my 'parse' script in blocks, as I need each set of results to be read into memory so that I can do index operations on them. For some reason, when I load the entire file into memory, I get more parsing results than if I use the following code to chop my input file into blocks:

with open(infile, 'r') as rfile:
    block = []
    thresh = 100000
    for i, line in enumerate(rfile):

        if i >= thresh:
            if "Iteration: 1" in line: # This marks the end of one set of results, and the beginning of the next, so we don't truncate any results
                read_block(block)
                thresh += 100000
                del block[:]

        block.append(line)

Any idea why I am losing data with this code? Or is everything kosher here, and my error is the result of this function interacting weirdly with the read_block() method...

Answer 1

This nested if is probably causing you problems (possibly with help from read_block ):

if i >= thresh:
    if "Iteration: 1" in line:
        read_block(block)
        thresh += 100000
        del block[:]

It is equivalent to a compound if condition, since it only runs the innermost block if both conditions are true:

if i >= thresh and "Iteration: 1" in line:
    read_block(block)
    thresh += 100000
    del block[:]

So whenever this loop encounters a short set of results --- less than 100,000 lines --- the outer for loop continues slurping in results sets until their combined length passes the threshold. When read_block is finally called, it will be given a block of lines that contains two or more sets. Can read_block cope with that?

Also, if you have a particularly long results set --- or any combination of sets that don't add up to exactly 100,000 lines --- the threshold is completely ignored until the loop reaches the end of the current set. If your other functions assume that block can never exceed 100,000 lines, they will get a rude surprise.

Finally, thresh is always incremented by 100,000, instead of however many lines were actually read. Since this can't happen until i >= thresh (possibly much greater-than), thresh will lag farther and farther behind reality. If that 100,000-line threshold really is important, you should set it to 100,000 lines from now :

thresh = i + 100000

To reiterate what I commented above, why not just feed read_block (or read_one_results_set or whatever) a single results set at a time, and not worry about counting lines?