如何删除 Linux 文本文件中每个（相同）行的 1 个实例？

Question

There is a file:

Mary 
Mary 
Mary 
Mary 
John 
John 
John 
Lucy 
Lucy 
Mark

I need to get

Mary 
Mary 
Mary 
John 
John 
Lucy

I cannot get the lines ordered according to how many times each line is repeated in the text, ie the most frequently occurring lines must be listed first.

Answer 1

If your file is already sorted (most-frequent words at top, repeated words only in consecutive lines) – your question makes it look like that's the case – you could reformulate your problem to: "Skip a word when it is encountered for the first time". Then a possible (and efficient) awk solution would be:

awk 'prev==$0{print}{prev=$0}'

or if you prefer an approach that looks more familiar if coming from other programming languages:

awk '{if(prev==$0)print;prev=$0}'

---

Partially working solutions below. I'll keep them for reference, maybe they are helpful to somebody else.

If your file is not too big, you could use awk to count identical lines and then output each group the number of times it occurred, minus 1.

awk '
{ lines[$0]++ }
END {
  for (line in lines) {
    for (i = 1; i < lines[line]; ++i) {
      print line
    }
  }
}
'

Since you mentioned that the most frequent line must come first, you have to sort first:

sort | uniq -c | sort -nr | awk '{count=$1;for(i=1;i<count;++i){$1="";print}}' | cut -c2-

Note that the latter will reformat your lines (eg collapsing/squeezing repeated spaces). See Is there a way to completely delete fields in awk, so that extra delimiters do not print?

Answer 2

don't sort for no reason:

 nawk '_[$-__]--' gawk '__[$_]++' mawk '__[$_]++'

Mary 
Mary 
Mary 
John 
John 
Lucy 

for 1 GB+ files, u can speed things up a bit by preventing FS from splitting unnecessary fields

mawk2 '__[$_]++' FS='\n'

for 100 GB inputs, one idea would be to use parallel to create, say, 10 instances of awk , piping the full 100 GB to each instance, but assigning each of them a particular range to partition on their end

(eg instance 4 handle lines beginning with FQ , etc), but instead of outputting it all THEN attempt to sort the monstrosity, what one could do is simply have them tally up, and only print out a frequency report of how many copies ( "Nx" ) of each unique line ( "Lx" ) has been recorded.

From there one could sort a much smaller file along the column holding the Lx 's, THEN pipe it to one more awk that would print out Nx # copies of each line Lx .

probably a lot faster than trying to sort 100 GB

I created a test scenario by cloning 71 shuffled copies of a raw file with these stats:

 uniq rows = 8125950. | UTF8 chars = 160950688. | bytes = 160950688.

 —- 8.12 mn unique rows spanning 154 MB

……resulting in a 10.6 GB test file:

  in0: 10.6GiB 0:00:30 [ 354MiB/s] [ 354MiB/s] [============>] 100%            
  rows = 576942450. | UTF8 chars = 11427498848. | bytes = 11427498848.

even when using just 1 single instance of awk , it finished filtering the 10.6 GB in ~13.25 mins - reasonable given the fact it's tracking 8.1 mn unique hash keys.

  in0: 10.6GiB 0:13:12 [13.7MiB/s] [13.7MiB/s] [============>] 100%            
 out9: 10.5GiB 0:13:12 [13.6MiB/s] [13.6MiB/s] [<=> ]

 ( pvE 0.1 in0 < testfile.txt | mawk2 '__[$_]++' FS='\n' )

  783.31s user 15.51s system 100% cpu 13:12.78 total


  5e5f8bbee08c088c0c4a78384b3dd328  stdin