在gzip文件Perl中读取和查找

Question

I am trying to read given set of gzip/plain xml files and printing some portions of these files into output xml files based on given offset and length values.

The offset values are keys of hash %offhash and corresponding keys are length.

Here is the funcntion I used for generating output files-

sub fileproc {
   my $infile = shift;
   my $outfile = shift;
   my $FILEH;
   $| = 1;
    $outfile =~ s/.gz$//;
   if($infile =~ m/\.gz$/i){
       open( $FILEH,"gunzip -c $infile | ") or die "Could not open input $infile";
   }
   else{
       open( $FILEH, "<", $infile) or die "Could not open input $infile";
   }

   open(my $OUTH, ">", $outfile) or die "Couldn't open file, $!";
   foreach my $offset (sort{$a <=> $b} keys %offhash)
   {
       my $record="";
       seek ($FILEH, $offset, 0);
       read ($FILEH, $record, $offhash{$offset}, 0);
       print $OUTH "$record";
   }
    close $FILEH;
    close $OUTH;
}

This function works properly for plain xml input files but creating some buffering issue when there are some(or all) .xml.gz files in the input file set. The output file in this case contains data from some previous read input( .gz) files.

It seems the problem is in the line--

open( $FILEH,"gunzip -c $infile | ") or die "Could not open input $infile";

Can anyone help me to resolve this issue?

Thanks in advance.

Answer 1

You can only seek in regular files, not in the output of programs or STDIN etc. If you want to do this, you need to add a buffering layer yourself, but note that you might to need to buffer the whole uncompressed file just to be able to seek in it.

Even if you don't gunzip with an external program, but use something like IO::Gzip you will not be able to seek, because the inherent way gzip (and other compressions) work, is that you need to read all the previous data to be able to decompress the data at the current file position. There are ways around it to limit the amount of necessary previous data, but then you would need to specifically prepare your gzip file and it will grow bigger. I'm not aware of any module which implements this currently, but I did a proof-of-concept once so I know it works.