How to troubleshoot very slow perl script that runs faster on one server but slow on another

Question

I received a perl script that is supposed to filter probes for Hi-C data, but it runs sooooooooo slow (several days). The one who gave me the script told me it only took a few hours on her previous lab's server, so I am wondering what would cause such a difference in run time, and how would I go about making it run faster? My knowledge of perl is very limited, but my google search so far suggests I may need to change some perl configuration files? Below is the main function in the whole script that takes the longest, the input file contains chromosome targets and locations. Any help would be greatly appreciated!

I have tried running with perl 5.16 and 5.26 to see if maybe the version was too old, but it took just as long.

sub get_restriction_fragments_for_targets {

    my @targets = @_;

    # Split by chromsome and sort
    my %targets;

    foreach my $target (@targets) {
            push @{$targets{$target->{chr}}},$target;
    }

    foreach my $chr (keys %targets) {
            my @sorted = sort {$a -> {start} <=> $b -> {start}} @{$targets{$chr}};

            # We also need to merge overlapping capture regions

            my @merged;

            my $last_region;

            foreach my $region (@sorted) {
                    unless ($last_region) {
                            $last_region = $region;
                            next;
                    }

                    if ($region->{start} < $last_region -> {end}) {
                            # Merge
                            if ($region ->{end} > $last_region->{end}) {
                                    $last_region->{end} = $region->{end};
                            }
                            next;
                    }

                    push @merged,$last_region;
                    $last_region = $region;
            }

            push @merged,$last_region;

            $targets{$chr} = \@merged;
    }

    my @target_fragments;

    open (IN,'all_restriction_fragments.txt') or die $!;

    my $last_chr = "";
    my $last_index = 0;

    while (<IN>) {
            chomp;
            my $line = $_;

            my ($chr,$start,$end) = split(/\t/);

            if ($chr ne $last_chr) {
                    warn "Moving to $chr\n";
                    $last_chr = $chr;
                    $last_index = 0;
            }

            next unless (exists $targets{$chr});

            my @local_targets = @{$targets{$chr}};

            foreach my $index ($last_index .. $#local_targets) {
                    my $target = $local_targets[$index];

                    if ($target -> {end} < $start) {
                            $last_index = $index;
                            next;
                    }

                    if ($target -> {start} > $end) {
                            last;
                    }

                    push @target_fragments,{
                            id => $target->{id},
                            chr => $chr,
                            start => $start,
                            end => $end,
                    };
                    last;
            }
    }

    return @target_fragments;
}

Answer 1

One can't tell without knowing a lot more about the problem, in the first place about the size of data structures and of the file that's processed.

Here is a generic comment on what would in principle be most time consuming operations in what the code seems to be doing, in a (probably) decreasing order of importance and/or likelihood

• How big is the file? That's disk access and that can take any time

• Sorting in a loop -- how big is data that is getting sorted? A sort is time consuming

• There's that "tap dance" around @sorted , which results in data copy -- how much data?

• In general, there is lots of data copying -- how large is @targets passed into the routine?

As you can see, each factor comes with the question of how large the data is. Disk access is of course costly, but a lot of data copying between various data structures in the program is just as important.

So if you can provide some detail that'll help us give you a more detailed analysis/guess.