从已排序的超大文件（每个200G）列表中删除重复项的最佳方法？

Question

• Other previously asked questions did not answer my question!

I have a series of large files (200 G ) each and each file is sorted and contain duplicates which look like this:

 50.21.180.100|a.ac
 50.21.180.100|a.ac
 50.21.180.100|a.ac
 50.21.180.100|a.ac
 50.21.180.100|a.ac
 50.21.180.100| b.ac
 50.21.180.100| b.ac
 50.21.180.100|b.ac
 50.21.180.100|b.ac
 50.21.180.100|b.ac
 50.21.180.100| c.ac
 50.21.180.100| c.ac
 50.21.180.100|c.ac
 50.21.180.100|c.ac
 50.21.180.100|c.ac
 50.21.180.100|c.ac
 50.21.180.100| d.ac

Expected output :

50.21.180.100|a.ac
50.21.180.100|b.ac
50.21.180.100|c.ac
50.21.180.100|d.ac

Does any body have any suggestion of the most optimal way (time and memory wise) of removing these duplicates? wether it is with Linux bash or Python or other languages?

Answer 1

首先删除空间，然后运行uniq：

cat infile.txt | tr -d " " | uniq > outfile.txt