Perl: STDOUT/the output of shell command to an array directly

Question

I have to access a shell command - hive within a Perl script, So I use `...`. Assuming the result of `hive ... ...` contains 100000000 lines and is 20GB size. what I want to achieve is like this:

@array = `hive ... ...`;

Does `` automatically know to use "\\n" as separator to divide each line into the @array?

The 2 ways I can thought of are (but with problem in this case):

$temp = `hive ... ...`;
@array = split ( "\n", $temp );
undef $temp;

The problem of this way is that if the output of hive is too big in this case, the $temp cant store the output, resulting in segmentation fault core dump.

OR

`hive ... ... 1>temp.txt`;
open ( FP, <, "temp.txt" );
while (<FP>)
{
    chomp;
    push @array, $_;
}
close FP;
`rm temp.txt`;

But this way would be too slow, because it writes result first to hard-disk.

Is there a way to write the output of a shell command directly to an array without using any 'temporary container'?

Very Thanks for helping.

Answer 1

@array = `command`;

does, in fact, put each line of output from command into its own element of @array . There is no need to load the output into a scalar and split it yourself.

But 20GB of output stored in an array (and possibly 2-3 times that amount due to the way that Perl stores data) will still put an awful strain on your system.

The real solution to your problem is to stream the output of your command through an IO handle, and deal with one line at a time without having to load all of the output into memory at once. The way to do that is with Perl's open command:

open my $fh, "-|", "command";
open my $fh, "command |";

The -| filemode or the | appended to the command tells Perl to run an external command, and to make the output of that command available in the filehandle $fh .

Now iterate on the filehandle to receive one line of output at a time.

while (<$fh>) {
    # one line of output is now in $_
    do_something($_);
}
close $fh;