如何使两个数据文件之间的差异更有效（运行时）

Question

I have a code which compares values on some specific terms between two files. The main time-consuming part of the code as follows:

my @ENTIRE_FILE;
my %NETS;
my %COORDINATES;
my $INT=1;
my %IR_VALUES;
################################# READING
foreach my $IR_REPORT_FILE_1(@IR_REPORT_FILES){
   {
      open (FHIN, "<", $IR_REPORT_FILE_1) or die("Could not open $! for reading\n");
      # chomp(my @ENTIRE_FILE = <FHIN>);                            # READS THE ENTIRE FILE
      local undef $/;
      @ENTIRE_FILE = split(/\n(.*NET.*)/,<FHIN>);
      close (FHIN);
   }
   ############################### BUILDING HASH
   for my $i(1..$#ENTIRE_FILE/2){
     if($ENTIRE_FILE[$i*2-1]=~ /^----.*\s+"(\w+)"\s+/){
       my $net =$1;
       my @ir_values_of_net = split(/\n/,$ENTIRE_FILE[$i*2]);
       for my $val (@ir_values_of_net){
       push ( @{ $NETS{$INT}{$net} }, $val ) if ($val =~ /^r.*\s+m1|v1_viadg|v1_viabar|m2|ay_viabar|ay_viadg|c1\s+/); # NETS{1}{VDD}=array of values, NETS{1}{VSS}, NETS{1}{AVDD}
       }
     }
   }
   $INT++;                                          # For the next file:  NETS{2}{VDD}, NETS{2}{VSS}, NETS{2}{AVDD}
}
############################### COMPARISON
my $loop_count=0;
foreach my $net(keys %{ $NETS{1} }){
   print "net is $net\n";
   foreach my $file_1_net( @{ $NETS{1}{$net} }){
     my @sub_str_1 = split (' ', $file_1_net);
     foreach my $file_2_net ( @{ $NETS{2}{$net} } ){
       $loop_count++;
#        my @sub_str_1 = split (' ', $file_1_net);
       my @sub_str_2 = split (' ', $file_2_net);
       if(($sub_str_1[2] eq $sub_str_2[2])&&(($sub_str_1[3].$sub_str_1[4].$sub_str_1[5].$sub_str_1[6] eq $sub_str_2[3].$sub_str_2[4].$sub_str_2[5].$sub_str_2[6]) || ($sub_str_1[3].$sub_str_1[4].$sub_str_1[5].$sub_str_1[6] eq $sub_str_2[5].$sub_str_2[6].$sub_str_2[3].$sub_str_2[4]))){
         push (@{ $COORDINATES{$net}{X} },$sub_str_1[3],$sub_str_1[5]) if ($sub_str_1[3] && $sub_str_1[5]);
         push (@{ $COORDINATES{$net}{Y} },$sub_str_1[4],$sub_str_1[6]) if ($sub_str_1[4] && $sub_str_1[6]);
         my $difference=$sub_str_1[1]-$sub_str_2[1];
         if($sub_str_1[3]=~/^-/){
           push (@{ $MATCHED_RESISTORS{$net}{$sub_str_1[2].$sub_str_1[3].$sub_str_1[4].$sub_str_1[5].$sub_str_1[6]} }, $file_1_net,$file_2_net,$difference);
         }else{
           push (@{ $MATCHED_RESISTORS{$net}{$sub_str_1[2]."-".$sub_str_1[3].$sub_str_1[4].$sub_str_1[5].$sub_str_1[6]} }, $file_1_net,$file_2_net,$difference);
         }
         push (@{ $IR_VALUES{$net} }, $sub_str_2[1]);
         last;
       }
     }
   } 
   print max @{ $IR_VALUES{$net} };
   print "\nloop count is $loop_count\n";
   $loop_count = 0;
#    <>;  
}

I ran a profiler on the code. Below is the output on the above part of code:

Some statistics:

1. For my testcase, the outer-most foreach has 3 elements. Below is the number of matched elements for each iteration:

 element_1: 14 element_1: 316 element_1: 8 

2. The file sizes are 8.3 MB and 518.3KB.
3. Run time for the entire code is: 220s
4. My main concern is when I have a file size of 8.3MB each, and there are more numbers of matching between two files, the run-time is humongous eg 3 hours.

My question is really simple: How do I make my code run faster?

Sample Data File_1:

r6_2389         1.29029e-05     ay_viabar       23.076   57.755   22.628   57.755   4.5      0        0        3.68449e-06      -5.99170336965613
r6_2397         1.29029e-05     ay_viabar       22.948   57.755   22.628   57.755   4.5      0        0        3.68449e-06      -5.99170336965613
r6_2400         1.29029e-05     ay_viabar       22.82    57.755   22.628   57.755   4.5      0        0        3.68449e-06      -5.99170336965613
r6_2403         1.29029e-05     ay_viabar       22.692   57.755   22.628   57.755   4.5      0        0        3.68449e-06      -5.99170336965613
r6_971          1.3279e-05      c1              9.492    45.742   -0.011   46.779   0.001    9.5589   10       0.0508653

Sample Data File_2:

r6_9261         0.00206167      ay_viabar       23.076   57.755   22.628   57.755   4.5      0        0        0.0207546    
r6_9258         0.00206167      ay_viabar       22.948   57.755   22.628   57.755   4.5      0        0        0.0161057    
r6_9399         0.00206167      ay_viabar       22.82    57.755   22.628   57.755   4.5      0        0        0.0127128    
r6_9486         0.00206167      ay_viabar       22.692   57.755   22.628   57.755   4.5      0        0        0.0103186    
r6_1061         1.3279e-05      cb_pc_viadg     -6.696   44.157   -0.159   44.847   0.001    0        0        0   

Sample Output:

   r6_9261         0.00206167      ay_viabar       23.076   57.755   22.628   57.755   4.5      0        0        0.0207546    
    r6_9258         0.00206167      ay_viabar       22.948   57.755   22.628   57.755   4.5      0        0        0.0161057    
    r6_9399         0.00206167      ay_viabar       22.82    57.755   22.628   57.755   4.5      0        0        0.0127128    
    r6_9486         0.00206167      ay_viabar       22.692   57.755   22.628   57.755   4.5      0        0        0.0103186 

The sample output is basically pushed into another hash which is further processed. But building up this hash consumes about 90% of the total run-time as per the profiler.

Answer 1

OK, so my first thought is - you've a 3 deep loop, and that will always be inefficient. We can probably trade memory for a lot of speed there.

Assuming the 'bigger' file is 'sample_1', otherwise swap them.

In this example - sample_2 will consume memory proportionate to the number of rows - so we ideally want that to be the smaller file. You may need to swap the match/test around, depending on whether file1 cols 5,6,3,4 matches file2 or vice versa.

But hopefully this illustrates a useful concept for solving your problem, if not entirely solving it?

Something like this will do the trick:

#!/usr/bin/env perl

use strict;
use warnings;

my %is_match; 

open ( my $sample_1, '<', 'sample1.txt' ) or die $!;
open ( my $sample_2, '<', 'sample2.txt' ) or die $!;

#    first of all, column 2 , 3,4,5,6 should match between 2 files. 
# and then print out both matching lines from two files.     
#    column 3,4,5,6 from one file can match with column 5,6,3,4. 

while ( <$sample_2> ) { 
   my @row = split; 
   #insert into hash
   #this would be much clearer if the fields were named rather than numbered
   #I think. 
   $is_match{$row[3]}{$row[4]}{$row[5]}{$row[6]}++; 
   $is_match{$row[5]}{$row[6]}{$row[3]}{$row[4]}++; 
}

while ( <$sample_1> ) {
   my @row = split;
   #print the current line if it matches from the hash above.
   print if $is_match{$row[3]}{$row[4]}{$row[5]}{$row[6]};
}

Because this iterates each file once, it should be a lot faster. and because one of your files is small, then that's the one you should read first into memory.

With your sample data as provided, this gives you the desired output.

The first loop reads though the file, selects your interest fields and inserts them into a hash, based on your 4 keys.

And then it does so again for the other set of valid matching keys.

The second loop reads the other file, selects the keys and just checks if either combination exists in the hash. And prints the current line if it does.