如何使用其中一个的第一列作为索引来合并多个 .csv 文件（shell 脚本）

Question

How to merge multiple .csv files using the 1st column of one of them as an index (pref shell scripting - awk)

88 .csv files that look like this input files names ZBND19X.csv

==> ZBND19X.csv <==
Gene,ZBND19X(26027342 pairs)
ENSTGUG00000013338 (GAPDH),984.31862471
ENSTGUG00000000915,862.597795025
ENSTGUG00000006651 (ARPP19),845.045872644
ENSTGUG00000005054 (CAMKV),823.404021741
ENSTGUG00000005949 (FTH1),585.628487964

and ZBND22V.csv

==> ZBND39X.csv <==
Gene,ZBND39X(26558640 pairs)
ENSTGUG00000013338 (GAPDH),971.678203888
ENSTGUG00000005054 (CAMKV),687.81249397
ENSTGUG00000006651 (ARPP19),634.296191033
ENSTGUG00000002582 (ITM2A),613.756010638
ENSTGUG00000000915,588.002298061

output file name RPKM_all.csv

Gene,ZBND19X(26027342 pairs),ZBND39X(26558640 pairs)
ENSTGUG00000013338 (GAPDH),984.31862471,971.678203888
ENSTGUG00000000915,862.597795025,588.002298061
ENSTGUG00000006651 (ARPP19),845.045872644,634.296191033
ENSTGUG00000005054 (CAMKV),823.404021741,687.81249397
ENSTGUG00000005949 (FTH1),585.628487964,0
ENSTGUG00000002582 (ITM2A),613.756010638,0

Adding the 0 when there is no corresponding value found.

Answer 1

join can only work on two files at a time, here comes

awk to the rescue!

$ awk -F, 'FNR==1 {c++; h=h sep $2; sep=FS; next} 
                  {ks[$1]; a[$1,c]=$2} 
           END    {print h; 
                   for(k in ks) 
                     {printf "%s", k;
                      for(i=1;i<=c;i++) printf "%s", FS a[k,i]+0; 
                      print ""}}' files

disclaimier: only if the data can fit in memory, also the order will be lost but if important there are ways to handle it.

Explanation Conceptually creating a table (aka 2D array, matrix) and filling up the entries. THe rows are indexed by key and columns by file number. Since awk array is hashing the keys we treat header separately to stay in place. a[k,i]+0 is to convert missing elements to 0 .

Answer 2

The simple answer is 'join'.

You can use the join command to match on the first column ( by default ) as long as the files are sorted.

Don't forget to sort your files.

Did I mention you need to sort your files ;)? It's an easy mistake to make ( I've made that mistake plenty; hence the emphasis ).

sort  ZBND19X.csv >  ZBND19X.csv.sorted
sort ZBND39X.csv  > ZBND39X.csv.sorted
join -t, ZBND19X.csv.sorted ZBND39X.csv.sorted > RPKM_all.csv

Here's the contents of RPKM_all.csv after running above :

ENSTGUG00000000915,862.597795025,588.002298061
ENSTGUG00000005054 (CAMKV),823.404021741,687.81249397
ENSTGUG00000006651 (ARPP19),845.045872644,634.296191033
ENSTGUG00000013338 (GAPDH),984.31862471,971.678203888
Gene,ZBND19X(26027342 pairs),ZBND39X(26558640 pairs)

We can also look for rows that don't match like this:

$ join -v1 -t, ZBND19X.csv.sorted ZBND39X.csv.sorted | awk -F, -v OFS=, '{print $1,$2,0}'
ENSTGUG00000005949 (FTH1),585.628487964,0

$ join -v2 -t, ZBND19X.csv.sorted ZBND39X.csv.sorted | awk -F, -v OFS=, '{print $1,0,$2}'
ENSTGUG00000002582 (ITM2A),0,613.756010638

Now you can combine the whole thing:

sort  ZBND19X.csv >  ZBND19X.csv.sorted
sort ZBND39X.csv  > ZBND39X.csv.sorted
join -t, ZBND19X.csv.sorted ZBND39X.csv.sorted > RPKM_all.csv
join -v1 -t, ZBND19X.csv.sorted ZBND39X.csv.sorted | awk -F, -v OFS=, '{print $1,$2,0}' >> RPKM_all.csv
join -v2 -t, ZBND19X.csv.sorted ZBND39X.csv.sorted | awk -F, -v OFS=, '{print $1,0,$2}'  >> RPKM_all.csv

Answer 3

the awk code (awk -F, 'FNR==1 {c++; h=h sep $2; sep=FS; next} ): does anyone can do more explanation on this, the code doesn't print the header correctly, all the headers just jump to different rows and the first header is missing too

P21 P22 P24 P24

AamoA_EU022762 1 1 0 0 AamoA_EU099963 0 1 0 0