Perl使用单个文件打开句柄渲染多个csv文件？

Question

I have this code, where I want to handle multiple csv(currently its just one file) file and use perl to render its format before its shipped to a linux-box and replaces original file content using ssh connection. Here is the code

#!/usr/bin/perl -w
use strict;
# this is a csv which will contains IP addresses of one specific category for e.g malware.
my $seculert_qradar_list = "$seculert_dir/seculert.csv";

#ssh connection information 
my $qradar_console = '10.10.1.22';
my $qradar_ssh_key = "$seculert_dir/qr-id_dsa";
my $qradar_ssh_knownhosts = "$seculert_dir/known_hosts";

#################################################################################

# NOTE: this is the "OUT" file.
# 1 - Name
# 2 - Sub-Name
# 3 - IP Address
# 4 - is colour, deprecated
# 5 - database length, deprecated
# 6 - asset weight, deprecated
# 7 - an ID for the 'record' each unique name pair (first 2 columns) gets an ID
#################################################################################
my $source = 'BAD-IP-Addresses-LABEL';
my $type_description = 'honeypots-for-examnple';

# Based upon the format described above I want to render the csv as written in print OUT statement. This format is important, because the endsystem process the file (remotenet.conf) based upon the provided layout.
open(FP, ">>$seculert_qradar_list");
for my $line (<FP>) {
    my ($hostname, $ip, $something1, $something2) = split(/,/, $line);
    print OUT "$source $type_description $ip #FF0000 0 90  29\n";
}
close(FP);

# Here I just want the contents of modified csv to be written over remotenet.conf. This file is then processed through auto-deploy script by the system. The results get populated on front-end webserver.    
print "Sending to QRadar...\n";
# SSH To QRadar's Console and push out file + trigger update
`scp -i $qradar_ssh_key -o UserKnownHostsFile=$qradar_ssh_knownhosts -o StrictHostKeyChecking=no root\@$qradar_console:/store/configservices/staging/globalconfig/remotenet.conf .`;
`sed -i -e '/^SECULERT/d' remotenet.conf`;
`cat $seculert_qradar_list >> remotenet.conf`;
`scp -i $qradar_ssh_key -o UserKnownHostsFile=$qradar_ssh_knownhosts -o StrictHostKeyChecking=no remotenet.conf root\@$qradar_console:/store/configservices/staging/globalconfig/remotenet.conf`;

print "Cleaning up...\n";
# Remove our SECULERT list and the newly pushed out qradar conf
unlink($seculert_qradar_list); unlink ('remotenet.conf');

print "Deploying in QRadar...(takes time to complete)\n";
# QRadar magic
`ssh -i $qradar_ssh_key -o UserKnownHostsFile=$qradar_ssh_knownhosts -o StrictHostKeyChecking=no root\@$qradar_console /opt/qradar/upgrade/util/setup/upgrades/do_deploy.pl`;
print "Complete!\n\n";

What I'm interested to know and perhaps get some help from perl programmer that using one file handle can I open multiple files eg In my case I have something like this

1. virus.csv
2. bot.csv
3. malware.csv

Do i need to re-copy for loop code for each csv file with a different handle? The destination file remotenet.conf remains the same.

After correct rendering of for eg one csv file the remotenet.conf on web-ui would look something like

Virus
    10.10.2.1
     .......

Bot
    10.10.4.1
     ......

It would be great that multiple changes happen in one-go, with just one auto-deploy(see code at end). I hope I'm able to understand the problem. Please let me know if more clarification is required.

thanks

WHAT I WANT TO ACCOMPLISH

I want a dynamic code, where there are multiple csv ready for rendering kept inside one folder. The sequence is described as :-

1. read each csv file from folder.
2. converts it into the acceptable format.
3. Change two variables values based upon csv file name. For eg

For filename malware.csv

my $source = 'BAD-IP-Addresses-LABEL'; my $type_description = 'honeypots-for-examnple';

For bot.csv

my $source = 'bot-net'; my $type_description = 'top-10';

At end, copies the formated file along with contents into remotenet.conf through ssh.

Answer 1

You cannot read from a file opened in append >> mode. It seems you've just renamed the OUT filehandle to FP , but that isn't enough.

• The original code included a chdir $seculert_dir – this makes sense because now we don't have to prefix our filenames with the directory all the time (it's the equivalent to the cd command in the shell).
• The original code didn't exactly use best practices. I wrapped the scp and ssh commands with versions that do a minimal amount of error checking.
• It is also unnecessary to invoke external tools for things we can do with Perl.

Here is my updated code, that can read the data from simplistic CSV files:

---

First, we start the preample. use warnings is more preferable than the -w switch.

#!/usr/bin/perl
use strict;
use warnings;

my $seculert_dir = "!FIXME!";
chdir $seculert_dir;  # go into that dir, so that we don't have to use prefixes

my @csv_files = ("seculert.csv"); # this is a csv which will contains IP addresses
                                  # of one specific category for e.g malware.
                                  # We only use the 2nd column

#ssh connection information 
my $qradar_console = '10.10.1.22';
my $qradar_ssh_key = "qr-id_dsa";
my $qradar_ssh_knownhosts = "known_hosts";

That was our configuration. Next we fetch the conf from the server using the wrapped SCP command:

# fetch the remotenet.conf from QRadar
print STDERR "Fetching configuration from QRadar...\n";
scp("root\@$qradar_console:/store/configservices/staging/globalconfig/remotenet.conf" => '.');

We now open a file where we put our changed configuration

# write the changed conf here before uploading:
open my $new_conf, ">", "remotenet.conf.changed" or die qq(Can't open "remotenet.conf.changed" for writing: $!);

Now we open the old configuration, copy it over, but skip lines starting with SECULERT .

# copy old conf over, delete lines starting with "SECULERT"
open my $old_conf, "<", "remotenet.conf" or die qq(Can't open "remotenet.conf" for reading: $!);
while (<$old_conf>) {
    print {$new_conf} $_ unless /^SECULERT/;
}
close $old_conf;

Notice how I've used “lexical filoehandles” ( open my $fh, ... ). This avoids some problems, and is more modern than using barewords.

Next, we loop through the CSV files. We open each one, then extract the 2nd column and print it with the other stuff into the changed configuration file.

# append the data from the CSVs
for my $csv_file (@csv_files) {
    my $source = 'BAD-IP-Addresses-LABEL';
    my $type_description = 'honeypots-for-examnple';

    open my $csv, "<", $csv_file or die qq(Can't open "$csv_file" for reading: $!);

    while (my $line = <$csv>) {
        my (undef, $ip) = split /,/, $line; # we're only interested in the 2nd column

        # Based upon the format described below I want to render the csv
        # as written in print OUT statement. This format is important, because the
        # endsystem process the file (remotenet.conf) based upon the provided layout.
        #
        # Columns in the output:
        # 1 - Name
        # 2 - Sub-Name
        # 3 - IP Address
        # 4 - is colour, deprecated
        # 5 - database length, deprecated
        # 6 - asset weight, deprecated
        # 7 - an ID for the 'record' each unique name pair (first 2 columns) gets an ID
        print {$new_conf} "$source $type_description $ip #FF0000 0 90  29\n";
    }
}

Now we have all the information we want in the new configuration file and can upload it to the server:

close $new_conf;

# copy the changed remotenet.conf back to QRadar
scp('remotenet.conf.changed' => "root\@$qradar_console:/store/configservices/staging/globalconfig/remotenet.conf");

# Remove our SECULERT list and the newly pushed out qradar conf
print STDERR "Cleaning up...\n";
unlink $_ or warn qq(Can't remove "$_": $!) for 'remotenet.conf', 'remotenet.conf.changed';

Next, we run the deploy script:

# QRadar magic -- run deploy script
print STDERR "Deploying in QRadar...(takes time to complete)\n";
ssh("root\@$qradar_console", '/opt/qradar/upgrade/util/setup/upgrades/do_deploy.pl');

print STDERR "Complete!\n\n";

Here are the wrappers for scp and ssh . They are subroutines (functions, procedures, or methods in other languages). The arguments are in the @_ array from which we unpack them into variables with better names. The system command takes a name of a command and a list of arguments. Because this circumvents the shell, we don't have to think about shell escapes. system returns zero on sucess, so we use that to check for errors.

# Wrappers for SSH and SCP commands
sub scp {
    my ($from, $to) = @_;
    my $success = 0 == system 'scp',
        '-i' => $qradar_ssh_key,
        '-o' => "UserKnownHostsFile=$qradar_ssh_knownhosts",
        '-o' => "StrictHostKeyChecking=no",
        $from => $to;
    return $success if defined wantarray;                # return failure when somebody checks for it
    die qq(Can't scp "$from" to "$to") if not $success;  # die when failure, and nobody checks.
}

sub ssh {
    my ($host, $command) = @_;
    my $success = 0 == system 'ssh',
        '-i' => $qradar_ssh_key,
        '-o' => "UserKnownHostsFile=$qradar_ssh_knownhosts",
        '-o' => "StrictHostKeyChecking=no",
        $host, $command;
    return $success if defined wantarray;                           # return failure when somebody checks for it
    die qq(Can't ssh into "$host" for '$command') if not $success;  # die when failure, and nobody checks.
}

---

This code could still be improved, eg by using a proper CSV parser like Text::CSV , using better SSH bindings than simply wrapping command line programs, automated error checking with autodie , and a better handling of tempfiles.

It seems you want different $source values for different files. For this, we should use a more complex data structure than @csv_files – I'd use a hash of arrays, eg

my %csv_files = (
  'malware.csv' => ['BAD-IP-Addresses-LABEL', 'honeypots-for-examnple'],
  'bot.csv'     => ['bot-net', 'top-10'],
);

This is a dictionary that maps keys (here filenames) to values (here the contents of two columns). Instead of looping over entries in an array, we would now loop over the keys in this hash:

for my $csv_file (keys %csv_files) {
  my ($source, $type_description) = @{ $csv_files{$csv_file} };

  ...
}

The expression $csv_files{$csv_file} accesses the entry called $csv_file in the hash $csv_files . This entry contains an array reference as value. The @{…} around that converts the array reference to an array, which we can unpack with a list assignment my ($foo, $bar) = @array .