Perl thread : Array of hash issue

Question

I have an issue trying to use perl threads module (I'm not good at this obviously).
I read many topics and tried some "tricks" but nothing helped me.
My goal is to fulfill my @allMember array with several hashes (%memberInfo).

Here is a simplified version of my code:

#!/usr/bin/perl
use strict ;
use warnings ;
use threads ;
use threads::shared ;

my %memberInfo:shared ;
my @member:shared ;
my @allMember:shared ;

my @list = ( 'toto:1.1.1.1:111', 'tata:2.2.2.2:222', 'titi:3.3.3.3:333') ;

@member = @list ;

my $nbThread = 5 ;
for ( 1..$nbThread ) { threads -> create ( \&job_to_parallelize ) ; }
foreach my $thr ( threads -> list() ) { $thr -> join(); }

sub job_to_parallelize {

  while ( my $member = shift(@member)) {
    my ($name,$ip,$port) = split(/:/,$member) ;

    $memberInfo{ip} = $ip ;
    $memberInfo{name} = $name ;
    $memberInfo{port} = $port ;
    push(@allMember,\%memberInfo) ;
           
  }
}

If I insert a print Dumper(\%memberInfo); above push(@allMember,\%memberInfo); I have the information I want:

$VAR1 = {
          'ip' => '1.1.1.1',
          'name' => 'toto',
          'port' => '111'
        };
$VAR1 = {
          'ip' => '2.2.2.2',
          'name' => 'tata',
          'port' => '222'
        };
$VAR1 = {
          'ip' => '3.3.3.3',
          'name' => 'titi',
          'port' => '333'
        };

But if I try a print Dumper(\@allMember); at the end of my script (after my "job_to_parallelize" function), the information is only the last %memberInfo hash cloned 3 times:

$VAR1 = [
          {
            'ip' => '3.3.3.3',
            'name' => 'titi',
            'port' => '333'
          },
          {
            'ip' => '3.3.3.3',
            'name' => 'titi',
            'port' => '333'
          },
          {
            'ip' => '3.3.3.3',
            'name' => 'titi',
            'port' => '333'
          }
        ];

For Information, my @list array is (in my real script) fulfilled from a big file (thousands of hosts).
I put the hosts in @list, and then I create a copy in my shared @member array.

I hope I'm clear enough.

Where am I wrong? Thanks for your help on this subject.

Answer 1

You empty the @member array in the first thread by doing shift(@member) here:

while ( my $member = shift(@member)) {

such that the following threads just sees an empty array. Try the following modification:

use feature qw(say);
use strict;
use warnings;
use threads ;
use threads::shared ;
use Data::Dumper qw(Dumper);

my @allMember:shared;
my @member = ( 'toto:1.1.1.1:111', 'tata:2.2.2.2:222', 'titi:3.3.3.3:333') ;
my $nbThread = 5 ;
for ( 1..$nbThread ) { threads -> create ( \&job_to_parallelize ) ; }
foreach my $thr ( threads -> list() ) {
    $thr -> join();
}

sub job_to_parallelize {
  for my $member (@member) {
      my ($name,$ip,$port) = split(/:/,$member);
      {
          lock @allMember;
          my %memberInfo:shared = ( ip => $ip, name => $name, port => $port );
          push(@allMember,\%memberInfo);
      }
  }
}

print Dumper(\@allMember);

Output :

$VAR1 = [
          {
            'port' => '111',
            'ip' => '1.1.1.1',
            'name' => 'toto'
          },
          {
            'port' => '222',
            'ip' => '2.2.2.2',
            'name' => 'tata'
          },
          {
            'port' => '333',
            'ip' => '3.3.3.3',
            'name' => 'titi'
          },
          {
            'ip' => '1.1.1.1',
            'port' => '111',
            'name' => 'toto'
          },
          {
            'ip' => '2.2.2.2',
            'port' => '222',
            'name' => 'tata'
          },
          {
            'name' => 'titi',
            'port' => '333',
            'ip' => '3.3.3.3'
          },
          {
            'name' => 'toto',
            'port' => '111',
            'ip' => '1.1.1.1'
          },
          {
            'port' => '222',
            'ip' => '2.2.2.2',
            'name' => 'tata'
          },
          {
            'name' => 'titi',
            'ip' => '3.3.3.3',
            'port' => '333'
          },
          {
            'ip' => '1.1.1.1',
            'port' => '111',
            'name' => 'toto'
          },
          {
            'name' => 'tata',
            'ip' => '2.2.2.2',
            'port' => '222'
          },
          {
            'port' => '333',
            'ip' => '3.3.3.3',
            'name' => 'titi'
          },
          {
            'port' => '111',
            'ip' => '1.1.1.1',
            'name' => 'toto'
          },
          {
            'ip' => '2.2.2.2',
            'port' => '222',
            'name' => 'tata'
          },
          {
            'ip' => '3.3.3.3',
            'port' => '333',
            'name' => 'titi'
          }
        ];

Edit :

If you want to empty the @member variable (it was not clear to me whether you wanted that or not) you can try this instead:

use feature qw(say);
use strict;
use warnings;
use threads ;
use threads::shared ;
use Data::Dumper qw(Dumper);

my @allMember:shared ;
my $lockvar:shared;
my @member:shared = ( 'toto:1.1.1.1:111', 'tata:2.2.2.2:222', 'titi:3.3.3.3:333') ;

my $nbThread = 5 ;
for ( 1..$nbThread ) { threads -> create ( \&job_to_parallelize ) ; }
foreach my $thr ( threads -> list() ) {
    $thr -> join();
}

sub job_to_parallelize {
    while (@member) {
        lock $lockvar;
        my $member = shift @member;
        my ($name,$ip,$port) = split(/:/, $member);
        {
            my %memberInfo:shared = ( ip => $ip, name => $name, port => $port );
            push(@allMember,\%memberInfo);
        }
    }
}

print Dumper(\@allMember);

Output :

$VAR1 = [
          {
            'name' => 'toto',
            'port' => '111',
            'ip' => '1.1.1.1'
          },
          {
            'name' => 'tata',
            'port' => '222',
            'ip' => '2.2.2.2'
          },
          {
            'name' => 'titi',
            'port' => '333',
            'ip' => '3.3.3.3'
          }
        ];