查找文件中的所有重复模式

Question

I have a file that contain a set of a few thousand unique words/terms. It looks like:

high school teacher
high school student
library
pencil stand
college professor
college graduate

I need the list of all repeated patterns, so in this case I would need the following as the result:

high
school
high school
college

Is there any way in unix/vim we could achieve this?

Additional elaboration on requirement:

Q. Do the repeats have to be on a single line, or can they be split over several lines?

• Ideally, each pattern should be in a new line

Q. Are the patterns all word sequences (one or more words)

• Yes they are all word sequences

Q. Does spacing matter within a line? Capitalization? Punctuation?

• spaces and punctuations are all counted as part of the pattern. We can ignore capitalisation

ie.

• School == School != school
• this pat.tern == this pat.tern != this pattern

Answer 1

This works for me (script placed in a file script.awk ):

{
    for (i = 1; i <= NF; i++)
    {
        count[$i]++
        sequence = $i
        for (j = i + 1; j <= NF; j++)
        {
            sequence = sequence " " $j
            count[sequence]++
        }
    }
}
END {
    for (i in count)
    {
        if (count[i] > 1)
           print i
    }
}

The 'every line' code builds up the word sequences on the line and uses those to count the sequences. The END block loops through the sequences, printing those with a count of more than one (so the word sequence was repeated).

Given the (extended) data file (called data ):

high school teacher
high school student
library
pencil stand
college professor
college graduate
coelacanths are ancient fish
coelacanths are ancient but still alive
coelacanths are ancient and long lived
coelacanths are ancient and can live to be 100 years old
coelacanths are ancient living fossils
coelacanths can live to be ancient
coelacanths are long-lived
coelacanths are slow to mature
coelacanths are denizens of the deep sea
coelacanths can be found off Africa and Indonesia

The output of awk -f script.awk data | sort awk -f script.awk data | sort is:

ancient
ancient and
and
are
are ancient
are ancient and
be
can
can live
can live to
can live to be
coelacanths
coelacanths are
coelacanths are ancient
coelacanths are ancient and
coelacanths can
college
high
high school
live
live to
live to be
school
to
to be

The data carefully has some longer repeated sequences of up to four words; longer word sequences would be tracked just as effectively.