Delete Perl Hash Values if duplication found

Question

I need a quick help , I'm greping some commands in a Unix Server and creating a hash out of those commands.

Issue is most of the time , there are duplicate Values which are coming and I want to remove any duplicate value of out these Hash.

Here is an example :

[randy@server04 ~/scripts]$ perl snmpperl.pl 
$VAR1 = {
    '1b' => [
        'abc_pl',
        'abc_pl',
        'abc_pl',
        'xyz_pl',
        'xyz_pl',
    ],
    '1a' => [
        'abc_pl',
        'abc_pl',
        'abc_pl',
        'abc_pl',
        'xyz_pl',
        'xyz_pl',
    ]

I need the Hash to be :-

$VAR1 = {
    '1b' => [
        'abc_pl',
        'xyz_pl',
    ],
    '1a' => [
        'abc_pl',
        'xyz_pl',
    ]

Answer 1

This is a relatively common Perl idiom, and is actually addressed in the FAQ, which you could locate by typing perldoc -q duplicate on any system with Perl installed.

Here is an adaptation on the ideas expressed in the FAQ:

use strict;
use warnings;
use Data::Dumper;

my %hash = (
  '1b' => [ 'abc_pl', 'abc_pl', 'abc_pl', 'xyz_pl', 'xyz_pl', ],
  '1a' => [ 'abc_pl', 'abc_pl', 'abc_pl', 'abc_pl', 'xyz_pl', 'xyz_pl', ],
);

foreach my $v ( values %hash ) {
  my %seen;
  @$v = grep { !$seen{$_}++ } @$v;
}

print Dumper \%hash;

This works by keeping track of whether any given element in the sub-array for a given hash key has been seen before. If not, pass it through the grep filter. Otherwise, don't send it through. In the end, all that gets built into the new structure are single instances of the array elements.

One nuance worth mentioning; The "it" variable in a foreach loop becomes an alias to the element it is representing. So in this case, for each iteration of the loop, $v aliases a hash element who's value contains an anonymous array reference. We simply replace contents of the anonymous array ref with the de-duped elements.

Answer 2

use List::MoreUtils 'uniq';

@$_ = uniq @$_ for values %hash;

Replacement for uniq from List::MoreUtils

sub uniq (@) {
    my %seen;
    grep !$seen{$_}++, @_;
}