在文件/数组中，搜索散列键，并将其替换为散列值，对所有散列键/值执行此操作

Question

I've searched around the site and surprisingly I can't seem to find something that will work for my particular problem. So I figured I'd post it and see how some of you more experienced programmers can address with problem.

I have a spreadsheet like text file (many lines with tab delimited columns), that I would like to search through for certain labels (ex scaffold1253.1_size81005.6.32799_7496) and replace them with more simplified labels (ex scaffold1253.1a). These labels are only in the first column of the text file. I've already written the script such that I have a hash with the old labels as keys corresponding to the new labels as their respective values. This hash has about 26000 lines. So essentially I'd like to take the hash keys 1 by 1, search for them in the text file, and replace them with their respective hash values.

I have a pretty good server availible so if its too complicated to make it first column specific to speed up the process then thats ok.

THis is what I have so far:

 use warnings;  



$gtf = './Hc_genome/Hc_rztk_1+2+8+9.augustus.gtf'; 
    open(FASTAFILE2, $gtf);
    @gtfarray = <FASTAFILE2>;
    #print @gtfarray;


my %hash;
while (<>)
{
   chomp;
   my ($key, $val) = split /\t/;
   $hash{$key} .= exists $hash{$key} ? ",$val" : $val;
}

#print %hash;

while (my ($find, $replace) = each %hash) {
    foreach (@gtfarray){
        $_ =~ s/$find/$replace/g;
        push @newgtf, $_;   
    }
}
print @newgtf;

This code doesn't seem to work as it doesn't complete. I'm pretty sure it's a problem with the foreach loop structure. Sorry I don't know of any other way to do this. Does anyone have a better way to run through this file and conduct the replacement?

Any input would be greatly appreciated! Thanks,

Andrew

@DVK

Here is the full script with your mods that runs into syntax errors with your while loop, any idea why it's not accepting it? Thanks again!

use warnings;  

$gtf = './Hc_genome/Hc_rztk_1+2+8+9.augustus.gtf'; 
    open(FASTAFILE2, $gtf);

my %hash;
while (<>){
    chomp;
    my ($key, $val) = split /\t/;
    $hash{$key} .= exists $hash{$key} ? ",$val" : $val;
}


while $line (<FASTAFILE2>){
    my @fields = split(/\t/, $line);
    # If you only care about first column, don't need the foreach loop below;
    #    just do the loop insides on $fields[0]
    foreach my $field (@fields) {
        $field = $hash{$field} if exists $hash{$field};
        print $outfile "$field\t"; # Small bug - will print training \t
    }
    print $outfile "\n"
}

__END__

Here is the syntax error: perl gtf_mod2.pl <./Hc_genome/header_file.txt syntax error at gtf_mod2.pl line 14, near "while $line " syntax error at gtf_mod2.pl line 23, near "}" Execution of gtf_mod2.pl aborted due to compilation errors.

Answer 1

You exhaust your file the first time through your loop using the initial $find and $replace key/value pair.

There are two potential solutions:

1. Open the file for reading during each iteration of your while loop (expensive)
2. Move the foreach loop to the outside of the while and iterate the hash each time (less expensive)

example:

REPLACE:
for my $line (@gtfarray) {
   while(my ($find, $replace) = each %hash) {
      if($line =~ s/$find/$replace/g) {
         push @newgtf, $line;
         next REPLACE; # skip to next iteration
      }
   }
   # if there was no replacement, push the old line
   push @newgtf, $line
}  

Answer 2

How big is the file that you are replacing the first column in?

If it's >50,000 lines, you are better off doing the reverse :

• Iterate through hash file once, and store that hash in memory

• Iterate through main file once , and for every line, for every column, find that value in the memorized hash, replace with hash value if found, and write.

In other words, remove the first @gtfarray = <FASTAFILE2>; and replace your last while loop with:

while my $line (<FASTAFILE2>) {
    my @fields = split(/\t/, $line);
    # If you only care about first column, don't need the foreach loop below;
    #    just do the loop insides on $fields[0]
    foreach my $field (@fields) {
        $field = $hash{$field} if exists $hash{$field};
        print $outfile "$field\t"; # Small bug - will print training \t
    }
    print $outfile "\n";
}

NOTE: I'm making an assumption that the fields contain FULL contents of your hash keys (eg your data file would contain a field with "scaffold1253.1_size81005.6.32799_7496" but NOT a field with "XYZscaffold1253.1_size81005.6.32799_7496___IOU").

If that assumption is wrong and you really DO need to run a regex because your scaffold strings may be contained in longer strings, there may still be a better solution aside from running O(N*M) regexes: if your scaffold strings are all of a certain well defined format (eg "scaffoldNNNNN.NNN_sizeNNNNN.NNN.NNNN_NNNN"), what you need to do then is:

• For each line of data file, run a single regex finding that pattern, with the entire pattern inside a capture group parenthesis:

   @matches = ($line =~ m/(scaffold\\d+\\.\\d+_size\\d+\\.\\d+\\.\\d+_\\d+/g ); 

• Then, look up every value of @matches array in the hash. If found, run ONLY the matches as as/// regex.

Answer 3

Could it be a job for Tie::File ? Assuming, that is, the data file could be operated on as an array.

use Tie::File; 

my $file = "./Hc_genome/Hc_rztk_1+2+8+9.augustus.gtf"; 

tie @lines, 'Tie::File', $file or die ;
for (@lines) {
 s/Oldlabel/NewLable/g;   # Change this to fit
}

untie @lines ;

Tie::File does a bunch of tricks to keep the "in place " changes to the file memory efficient.

Answer 4

Looking at your previous post , wouldn't it be more simple to create the shortened 'id' while reading the file. Then you would have no need of the other file where you get your hash?

Here is the (untested) code below. (would need to direct the print statements to an output file on the command line or open a file for writing in your script).

#!/usr/bin/perl
use strict;
use warnings;

my $gtf = './Hc_genome/Hc_rztk_1+2+8+9.augustus.gtf';
open my $FASTAFILE2, "<", $gtf or die "Unable to open '$gtf' for reading. $!";

my %seen;

while (<$FASTAFILE2>) {
    chomp;
    my ($id, $val) = split /\t/, $_, 2;

    # copy $id to $prefix and
    # remove everything after '.1' in $prefix
    (my $prefix = $id) =~ s/\.1\K.*//; 

    if ($seen{$id}) {
        ++$seen{$id};
    }
    else {
        $seen{$id} = 'a';   
    }
    print "$prefix$seen{$id}\t$val\n";
}

close $FASTAFILE2 or die "Unable to close '$gtf' from reading. $!";