如何加快此Perl脚本的grep速度

Question

Currently I have a script that need to go extraction logs. Below is the Perl code snippet: The script traverse every server folder and grep the necessary information. The problem is that when the number of logs maybe huge, the script may take very long time to finish. The bottle neck is this line:

@leaf_lines = qx($grep -l "stagename = $current_stage" $grep_path| xargs $grep "Keywords")

I am wondering if there is any way to speed up this operation? The script is running on a server with 8 cores per CPU and 8G memory, is there any way to use these resources?

my $grep = ($leaflog_zipped) ? "zgrep" : "grep" ;
my %leaf_info;
my @stage = ("STAGE1", "STAGE1", "STAGE3");
foreach my $leaf_dir (@leaf_dir_list){
    my $grep_path = $log_root_dir . "/$leaf_dir/*" ;          
    foreach my $current_stage (@stage){
        my @leaf_lines;
        @leaf_lines = qx($grep -l "stagename = $current_stage" $grep_path| xargs $grep "Keywords"); ## how to improve the grep speed?  
        foreach (@leaf_lines){
            if(...){
                $leaf_info{$current_stage}{xxx} = xxxx;
            }
        }    
    }
}

Answer 1

For starters - I'd say don't 'shell out' to grep - perl has perfectly good built in pattern matching and regular expressions, and includes the ability to precompile a regular expression.

http://perldoc.perl.org/perlop.html#Regexp-Quote-Like-Operators

Also - you can run perl in parallel fairly easily using threading or forks, which makes better use of your CPU resources.

However I will point out - things like grep aren't generally CPU related problems. CPUs are pretty fast these days, where filesystems are generally a lot slower. You will probably be spending more of your time reading data from disk than you will processing it, by quite a large margin.

So the thing that will likely be giving you a lot of grief is you grep multiple times.

my $grep_path = $log_root_dir . "/$leaf_dir/*" ;          
foreach my $current_stage (@stage)

Each element of @stage triggers another grep, and it's doing so for every file in that directory. And then you're grepping again .

That's a poor algorithm, because you'll be reading every file multiple times. Why not instead do something like:

#could do this with map - I haven't for clarity. 
my %stages;
$stages{'STAGE1'}++;
$stages{'STAGE2'}++;
$stages{'STAGE3'}++;

foreach my $file ( glob $grep_path ) {
    open( my $input_fh, "<", $file ) or die $!;
    while (<$input_fh>) {
        if (m/current_stage/) {
            my ($file_stage) = (
                m/stagename = (\w+)/;
            );
            if ( $stages{$file_stage} ) {
                # do something here
            }
        }
    }
}

That way - whilst you do have to read every file - you only do so once.

Answer 2

Yes, definitely. Simply replace xargs with GNU Parallel or another similar program (there are multiple programs named parallel on some Linux systems, so be mindful of which one you have; GNU Parallel is probably the best).