与正则表达式重叠匹配 ngrams

Question

I have a string and need to use regex.

"hello COMMA the matche's roll over matche's or the expression for details PCRE flavors of regex are supported here"

and i want to find bi and trigrams of it. So focusing on bigrams it should pull

hello COMMA
COMMA the
the matche's
etc

Ive written this regex to do that but its not grabbing the overlapping results.

[\w'-]+ [\w'-]+

it will only grab

hello COMMA
the matches
etc

when i wrap it in ?= like this it grabs all sorts of trash now. What am I missing?

(?=([\w'-]+ [\w'-]+))

also the overlap=True thing doesnt work for me for somereason

Answer 1

Do not use regular expressions for text processing. There is the package NLTK that was specifically designed for that job:

import nltk
text = "hello COMMA the matche's roll over ..."
words = nltk.word_tokenize(text)
list(nltk.bigrams(words))
# [('hello', 'COMMA'), ('COMMA', 'the'), ('the', 'matche'),...]
list(nltk.trigrams(words))
#[('hello', 'COMMA', 'the'), ('COMMA', 'the', 'matche'), ...]

Answer 2

Would you please try the following:

import re

str = "hello COMMA the matche's roll over matche's or the expression for details PCRE flavors of regex are supported here"

matches = re.finditer(r'\S+\s(?=(\S+))', str)
for match in matches:
    print(match.group(0) + match.group(1))

Output:

hello COMMA
COMMA the
the matche's
matche's roll
[snipped]

The regex (?=(\\S+)) includes a capture group within the positive lookahead assertion. It assigns match.group(1) to the matched substring without moving the position forward thanks to the zero-width matching.

Answer 3

The regular expression below is a generalisation and simplification of the regex suggested in a comment on the question by @Wiktor. Wiktor's solution was for 2-grams (or bigrams). This solution is for 3-grams (or trigrams). For n-grams, where n is a variable, replace {2} with {#{n-1}} .

First assume that the string contains only word characters and whitespace. The following regex can then be used to extract the trigrams:

(?=(?<!\S)(\w+(?:\s+\w+){2}))

Example

The regex can be broken down as follows:

(?=           # begin a positive lookahead   
  (?<!        # begin a negative lookbehind
    \S        # match a a non-whitespace char
  )           # end the negative lookbehind
  (           # begin capture group 1
    \w+       # match 1+ word chars
    (?:       # begin a non-capture group
      \s+\w+  # match 1+ whitespace chars followed by 1+ word chars
    )         # end non-capture group
    {1,2}     # execute the non-capture group 1-2 times 
  )           # end capture group
)             # end positive lookahead

If, as in the example, the string may also contain apostrophes within words (but not at the beginning or end of a word), each token \\w+ above can be replaced with \\w+(?:[']\\w+)* to obtain:

(?=(?<!\S)((?:\w+(?:[']\w+)*(?:\s+\w+(?:[']\w+)*){1,2})))

Example

The regex quickly breaks down, however, if too much is asked for the possible numbers and locations of certain characters.

This is an example of a situation where a regex should not be used, as the desired array can be produced much more easily with other tools. It is a useful exercise, however, as it does sharpens one's facility with regular expressions.