将竖线分隔的文本流中的列标准化为行

Question

I am looking for a concise, command line tool/script to take the output of another command and transform the text into a normalized data set for import to a database.

My input stream currently looks like this:

timestamp|identifier|column1|column2|...|column n

(representing n observations at the same time, for the same identifier (person) )

I want to grab the first two fields and then prepend them to column1-n to produce output like this:

timestamp|identifier|column1
timestamp|identifier|column2
timestamp|identifier|column3
...
timestamp|identifier|column n

sed? awk? perl? or, would it be better to load this data into a database table as-is, then use some kind of transform script stored procedure? I believe I've done this before in SQL Server using PIVOT

Answer 1

This can make it:

$ awk 'BEGIN{FS=OFS="|"} {for (i=3; i<=NF; i++) print $1, $2, $i}' file
timestamp|identifier|column1
timestamp|identifier|column2
timestamp|identifier|...
timestamp|identifier|column n

Explanation

• BEGIN{FS=OFS="|"} set input and output field separator as | .
• for (i=3; i<=NF; i++) print $1, $2, $i loop through all fields since 3rd printing 1st col, 2nd col + current field.

Answer 2

perl -F'\|' -lane 'print join "|", @F[0,1],$_ for @F[2..$#F]' file

output

timestamp|identifier|column1
timestamp|identifier|column2
timestamp|identifier|column n

Explanation:

-F'\\|' is delimiter for implicit split, and it should it be escaped since it is regex

-l auto chomp newline, and adds one when printing

-a auto split into @F array

-n adds implicit while(<>) loop

or letting perl speak for itself,

perl -MO=Deparse -F'\|' -lane 'print join "|", @F[0,1],$_ for @F[2..$#F]'
BEGIN { $/ = "\n"; $\ = "\n"; }
LINE: while (defined($_ = <ARGV>)) {
    chomp $_;
    our(@F) = split(/\|/, $_, 0);
    print join('|', @F[0, 1], $_) foreach (@F[2 .. $#F]);
}