逐行读取文件而不将整个文件加载到内存中

Question

I am working with a 50 Gb MySQL export file, and performing a list of scripting operations on it to convert to an SQLite3 loadable form (I got the leads from here : script to convert mysql dump sql file into format that can be imported into sqlite3 db ). I have not studied the structure of the MySQL dump, the data was got from a third party. I can see that it has create table and insert into statements, but given the size it is hard to manually read through and understand the structure. Piping the file through will not work because of the size. Also a bash script to load the file and then process line by line, such as

while read line
<do something>

complains that it is Out of Memory.

So I tried to pick each line, using awk or sed (both work), write the line to a file and then pass it through the list of perl scripts. This is the awk script I am using

$ awk -vvar="$x" 'NR==var{print;exit}' file > temp

where x holds the line number and then temp is sent through the perl commands and finally appended to the output file.

However, although fast initially, it quickly slows down as it starts having to iterate over an increasing number of lines from the start. There are about 40,000 lines.

Has anyone worked with something like this? Is there a faster way of doing this?

Answer 1

Simply process one line at a time:

while read -r line
do
    echo "$line" > temp
    …process temp with Perl, etc…
done < file

At least this won't exhibit quadratic behaviour reading the file, which is what your awk script does. It reads the big file exactly once, which is optimal performance in Big-O notation (within a constant factor).

If, as you say, that causes problems in bash , then you should use Perl to read each line. With 40,000 lines in 50 GiB of data, you have about 1¼ MiB of data per line. That is unlikely to cause Perl any problems, though it might perhaps cause Bash problems. You can either revise the existing Perl to read one line at a time, or use a simple wrapper Perl script that does the job of the Bash script above.

wrapper.pl

Assuming your current processing script is called script.pl :

#!/usr/bin/env perl
use strict;
use warnings;

my $file = "temp";

while (<>)
{
    open my $fh, ">", $file or die;
    print $fh $_;
    close $fh;
    system "perl", "script.pl", $file;
}

^{Untested code}

Invocation:

perl wrapper.pl <file >output