根据perl中的列值将大文件拆分为小文件

Question

I am trying to split up a large file (having around 17.6 million data) into 6-7 small files based on the column value.Currently, I am using sql bcp utility to dump in all data into one table and creating seperate files using bcp out utility.

But someone suggested me to use Perl as it would be more faster and you don't need to create a table for that.As I am not a perl guy. I am not sure how to do it in perl. Any help..

INPUT file :

inputfile.txt

0010|name|address|city|.........
0020|name|number|address|......
0030|phone no|state|street|...

output files:

0010.txt

0010|name|address|city|.........

0020.txt

0020|name|number|address|......

0030.txt

0030|phone no|state|street|...

Answer 1

It is simplest to keep a hash of output file handles, keyed by the file name. This program shows the idea. The number at the start of each record is used to create the name of the file where it belongs, and file of that name is opened unless we already have a file handle for it.

All of the handles are closed once all of the data has been processed. Any errors are caught by use autodie , so explicit checking of the open , print and close calls is unnecessary.

use strict;
use warnings;
use autodie;

open my $in_fh, '<', 'inputfile.txt';

my %out_fh;

while (<$in_fh>) {
  next unless /^(\d+)/;
  my $filename = "$1.txt";
  open $out_fh{$filename}, '>', $filename unless $out_fh{$filename};
  print { $out_fh{$filename} } $_;
}

close $_ for values %out_fh;

Note close caught me out here because, unlike most operators that work on $_ if you pass no parameters, a bare close will close the currently selected file handle. That is a bad choice IMO, but it's way to late to change it now

Answer 2

17.6 million rows is going to be a pretty large file, I'd imagine. It'll still be slow with perl to process.

That said, you're going to want something like the below:

use strict;
use warnings;

my $input = 'FILENAMEHERE.txt';
my %results;

open(my $fh, '<', $input) or die "cannot open input file: $!";
while (<$fh>) {
  my ($key) = split '|', $_;
  my $array = $results{$key} || [];
  push $array, $_;
  $results{$key} = $array;
}

for my $filename (keys %results) {
  open(my $out, '>', "$filename.txt") or die "Cannot open output file $out: $!";
  print $out, join "\n", $results{$filename};
  close($out);
}

I haven't explicitly tested this, but it should get you going in the right direction.

Answer 3

$ perl -F'|' -lane '
    $key = $F[0];
    $fh{$key} or open $fh{$key}, ">", "$key.txt" or die $!;
    print { $fh{$key} } $_
  ' inputfile.txt

Answer 4

perl -Mautodie -ne'
  sub out { $h{$_[0]} ||= open(my $f, ">", "$_[0].txt") && $f }
  print { out($1) } $_ if /^(\d+)/;
' file