逐行读取文件，超时时间太长？

Question

I have a 1.2TB file that I am running some code against, but constantly running into OutOfMemoryError exceptions. I ran the following two pieces of code against the file to see what was wrong:

import sys

with open(sys.argv[1]) as f:
    count = 1
    for line in f:
        if count > 173646280:
            print line
        else:
            print count
            count += 1

And this code:

#!/usr/bin/env perl
use strict;
use warnings;

my $count = 1;
while (<>) {
    print "$count\n";
    $count++;
}

Both of them zoom until they hit line 173,646,264, and then they just completely stop. Let me just give a quick background on the file.

I created a file called groupBy.json . I then processed that file with some Java code to transform the JSON objects and created a file called groupBy_new.json . I put groupBy_new.json on s3, pulled it down on another server and was doing some processing on it when I started getting OOM errors. I figured that maybe the file got corrupted when transferring to s3. I ran the above Python/Perl code on groupBy_new.json on both serverA (the server where it was originally at), and serverB (the server from which I pulled the file off s3), both halted at the same line. I ran then ran the above Python/Perl code on groupBy.json , the original file, and it also halted. I tried to recreate groupBy_new.json with the same code that I had used to originally create it, and ran into an OOM error.

So this is a really odd problem that is perplexing me. In short, I'd like to get rid of this line that is causing me problems. What I'm trying to do is read a file with a timeout on the line being read. If it cannot read the input line in 2 seconds or so, move on to the next line.

Answer 1

What you can do is count the number of lines until the problem line and output it - make sure you flush the output - see https://perl.plover.com/FAQs/Buffering.html . Then write another program that will copy the first of this number of lines to a different file, and then read the file's input stream character by character (see http://perldoc.perl.org/functions/read.html ) until it hits a "\\n" and then copy the rest of the file - either line by line or in chunks.