如果file1的第一列与file2中的任何字符串匹配，则将其替换为file1的第二列

Question

I have this problem and I haven't solved it... I would like to manipulate this files.. if the first column of file1 matches any string in file2, then replace it with the second column of file1... then collapse it (I mean,I need just unique values per field or "cell" in second column of the output_file)..
It doesn't matter which language solves this (awk, perl, python)... files contains 100000 lines or more... I've been trying one-line awk scripts, but nothing...

Any help appreciated.
Regards

file1.txt

ID100000360640  ITEM1;ITEM2  
ID100000360638  ITEM1;ITEM3  
ID100000360644  ITEM1;ITEM4  
ID100000363115  ITEM5;ITEM2;ITEM3  
ID100000363116  ITEM1;ITEM7  
ID100000382126  ITEM8;ITEM1  
ID100000002165  ITEM1;ITEM2;ITEM3;ITEM9  
ID100000002596  ITEM1;ITEM10  
ID100000003084  ITEM1  

file2.txt

ID200000000419  ID100000360638;ID100000360640;ID100000360644;ID100000394921
ID200000000938 ID100000363115;ID100000363116;ID100000363117;ID100000382126  
ID200000001036  ID100000002165;ID100000398119 

output_expected.txt

ID200000000419  ITEM1;ITEM3;ITEM1;ITEM2;ITEM1;ITEM4;ID100000394921  
ID200000000938  ITEM5;ITEM2;ITEM3;ITEM1;ITEM7;ID100000363117;ITEM8;ITEM1;  
ID200000001036  ITEM1;ITEM2;ITEM3;ITEM9;ID100000398119  

processed_output.txt

ID200000000419  ITEM1;ITEM2;ITEM3;ITEM4;ID100000394921  
ID200000000938  ITEM1;ITEM2;ITEM3;ITEM5;ITEM7;ITEM8;ID100000363117;  
ID200000001036  ITEM1;ITEM2;ITEM3;ITEM9;ID100000398119 

Thanks

Answer 1

Through python3.

#!/usr/bin/python3
with open('file1.txt') as f, open('file2.txt') as r:
    d = {}
    m = f.read()
    for line in m.split('\n'):
        try:
            d.update(dict([tuple(line.split())]))
        except:
            pass
    j = r.read()
    for k in d:
        j = j.replace(k, d[k])
    print(j)    

Output:

ID200000000419  ITEM1;ITEM3;ITEM1;ITEM2;ITEM1;ITEM4;ID100000394921
ID200000000938 ITEM5;ITEM2;ITEM3;ITEM1;ITEM7;ID100000363117;ITEM8;ITEM1  
ID200000001036  ITEM1;ITEM2;ITEM3;ITEM9;ID100000398119 

Answer 2

This produces the collapsed output:

$ awk 'FNR==NR{a[$1]=$2;next} {c="";delete d;delete e;split($2, b, /;/);for (i in b)c=c";"(a[b[i]]?a[b[i]]:b[i]);split(substr(c,2),d,/;/); for(i in d)e[d[i]]=1; c=""; for (i in e){c=c";"i}; print $1,substr(c,2)}' file1.txt file2.txt
ID200000000419 ID100000394921;ITEM1;ITEM2;ITEM3;ITEM4
ID200000000938 ITEM1;ITEM2;ITEM3;ID100000363117;ITEM5;ITEM7;ITEM8
ID200000001036 ITEM1;ITEM2;ITEM3;ID100000398119;ITEM9

How it works

• FNR==NR{a[$1]=$2;next}

  While we are reading the first file, this creates an associative array a which associates the first field as a key with the second as a value. Thus, the value of a[ID100000360640] is ITEM1;ITEM2 . This is done for all lines of file1.txt . The next statement causes all the remaining commands to be skipped and jumps to the next line.

• c="";delete d;delete e

  If we have gotten here, that means that we are working on the second file, file2.txt . These three commands initialize variable c and arrays d and e for the new line.

• split($2, b, /;/)

  This splits the second field on semicolons and assigns the result to array b .

• for (i in b)c=c";"(a[b[i]]?a[b[i]]:b[i])

  This creates the uncompressed output.

• split(substr(c,2),d,/;/); for(i in d)e[d[i]]=1

  This creates a associative array e whose keys are each of the fields in the uncompressed output.

• c=""

  This initializes c again to an empty line before we add to it the compressed output.

• for (i in e)c=c";"i

  For each key in array e , we add the key to string c . This creates the compressed output.

• print $1,substr(c,2)

  This prints the complete compressed line.

Answer 3

Reasonably short awk way

awk 'FNR==NR{a[$1]=$2;next}
     {for(i in a)gsub(i,a[i])
      x=split($2,b,";")
      for(i=1;i<=x;i++)y!~b[i]";"&&y=y?y";"b[i]:b[i];$2=y;y=""}1' file file2

---

Output

ID200000000419 ITEM1;ITEM3;ITEM2;ITEM4;ID100000394921
ID200000000938 ITEM5;ITEM2;ITEM3;ITEM1;ITEM7;ID100000363117;ITEM8
ID200000001036 ITEM1;ITEM2;ITEM3;ITEM9;ID100000398119

---

How it works

FNR==NR{a[$1]=$2;next}

When the File Record Number matches total Record Number(effectively means whilst reading the first file) assign the second field to an array using the first field as a key. Next means skip all further instructions and go to the next record.

for(i in a)gsub(i,a[i])

Now we are in the second file as FNR!=NR anymore.
For each element in the array gsub swaps everything that matches the key with what is contained in the array.

x=split($2,b,";")

Split the second field into array b separated by a ; .
Assign the size of the array to x.

for(i=1;i<=x;i++)

Loop from to the size of the array.

y!~b[i]";"&&  

if variable y already contains the split value in b then don't continue.

y=y?y";"b[i]:b[i] 

if y exists add the value in b[i] to the end or else just set y to b[i].

$2=y;y=""

Set second field to value in y(our new string) and reset y to nothing.

---

Resources

https://www.gnu.org/software/gawk/manual/html_node/String-Functions.html