使用Pandas或AWK合并2个不相等的文件

Question

I have 2 csv that need to be merged, File1 is around 20gb and other only has ~1000 lines. Because of the large size I first iterate over bigger file and want to filter the bigger file (say file1) to a smaller file and then load/merge using pandas

File1: The bigger file is as below

col1,col2,col3
1,1,1491795901021327897
1,2,1491795901021342873
1,3,1491795901021347247
1,4,1491795901021351620
1,5,1491795901021356612
1,6,1491795901021361172
1,7,1491795901021366797

The smaller file is as below

col1,col2,col3,col4,col5,col6
val1,val2,val3,1,6,1412414141412414
val1,val2,val3,1,3,1434252352352325

One way I did was to create a single key from both files by doing 10*10**10(value at col1) + val at col2 and similarly in smaller file using col4,5. saving this values as a list and for each line in bigger file if value is present in list print that row. So finally a small filtered file is printed. Is there a better way to do this in python or using awk maybe.

Ultimate intent is to merge, but since 20gb cannot be loaded in pandas so I'm filtering my file and making it smaller. I'm sure there must be a better way to approach this.

Answer 1

awk to the rescue!

by extrapolation, I think that's what you want

$ awk 'BEGIN        {FS=OFS=","} 
       NR==1        {h=$0; next} 
       NR==FNR      {a[$4,$5]=$0; next} 
       FNR==1       {print h,$3} 
       ($1,$2) in a {print a[$1,$2],$3}' small large

col1,col2,col3,col4,col5,col6,col3
val1,val2,val3,1,3,1434252352352325,1491795901021347247
val1,val2,val3,1,6,1412414141412414,1491795901021361172

It should be easy to read but I can write explanation if I get feedback on my interpretation.

Answer 2

Try this -

$ head f?
==> f1 <==
col1,col2,col3
1,1,1491795901021327897
1,2,1491795901021342873
1,3,1491795901021347247
1,4,1491795901021351620
1,5,1491795901021356612
1,6,1491795901021361172
1,7,1491795901021366797

==> f2 <==
col1,col2,col3,col4,col5,col6
val1,val2,val3,1,6,1412414141412414
val1,val2,val3,1,3,1434252352352325
$ awk -F, 'NR==FNR{a[$4 FS $5]=$6;next} ($1 FS $2) in a {print $0 FS a[$1 FS $2]}' f2 f1
1,3,1491795901021347247,1434252352352325
1,6,1491795901021361172,1412414141412414

Explained -

Created key using $4FS$5 from file f2 and matching it with key of $1FS$2 of file f1 , if $4FS$5 of f2 is matching with $1FS$2 then print all contents from file f1 along with the $6 from file f2 .