正则表达式不匹配字符串末尾的整个单词（bigram），仅在开头和中间

Question

I want to take a bigram and see if it is present in two segments (strings), called source and target (as in one source language being translated into a target language). For example, "star wars", is present in "star wars movie" and in "star wars filme". This means that "star wars" is untranslated. I am using a regular expression, so that the match is on whole words and not substrings. It is working for the two segments above, but it is not working when "star wars" is at the end of the second string, as in "filme star wars".

The bigram is read from a file that contains one bigram per line, and I am removing the newline at the end:

topword = input_file0.readline().lower().replace('\n', "")

The source and target segments are being read from files, and I am removing the newline at the end:

srcsegm = input_file1.readline().lower().replace('\n', "")
tgtsegm = input_file2.readline().lower().replace('\n', "")

The regex for the match is:

regex_match = re.compile(rf'\b{re.escape(topword)}\b')

I test if there is a match in the source:

has_match_src = re.search(regex_match,srcsegm)

If there is a match, I test for a match in the target:

has_match_tgt = re.search(regex_match,tgtsegm)

If both are true, I mark this as an "untranslated" term, because it is the same in source and target languages.

I am printing results to see what is happening, as:

print(topword,"untr:",srcsegm,"=====",tgtsegm)
print(topword,"translated:",srcsegm,"=====",tgtsegm)

But the results below are correct when "blu ray" is at the beginning or in the middle of the string:

blu ray  untr: blu ray rar ===== blu ray rar
blu ray  untr: soul blu ray disney ===== soul blu ray disney

And wrong when blu ray is at the end of the string:

blu ray  translated: sony blu ray player ===== sony odtwarzacz blu ray

It should say "untr" since we can see "blu ray" in the source segment and also in the target. The question is: why is it not producing a match at the end of the string?

This is the code:

topword = input_file0.readline().lower().replace('\n', "") # for ngrams, do not use strip and replace the newline
count_untr = 0
count_tr = 0

while len(topword)>0:   # looping the topword
    count_untr = 0
    count_tr = 0
    srcsegm = input_file1.readline().lower().replace('\n', "")
    tgtsegm = input_file2.readline().lower().replace('\n', "")
    regex_match = re.compile(rf'\b{re.escape(topword)}\b')
 
    while len(srcsegm)>0:     # looping the src and tgt segments for a topword
        has_match_src = re.search(regex_match,srcsegm) 
        if has_match_src != None:
            has_match_tgt = re.search(regex_match,tgtsegm)
            if has_match_tgt != None:
                count_untr += 1
                print(topword,"untr:",srcsegm,"=====",tgtsegm)
            else:
                count_tr += 1
                print(topword,"translated:",srcsegm,"=====",tgtsegm)

Thanks in advance.

Answer 1

If, for example, you were looking for 'star wars' within a string we wanted to allow arbitrary whitespace (such as a newline) between the two words and these words must appear on word boundaries, then the actual regex you would want to use would be:

\bstar\s+\wars\b

With that in mind you should be splitting topword into into its two component words and building a search regex by escaping each word individually and combing with whitespace between them:

import re

#topword =  input_file0.readline().lower().strip()
topword = 'star wars'

#srcsegm = input_file1.readline().lower().strip()
srcsegm = 'i just saw star wars and loved it!'

#tgtsegm = input_file2.readline().lower().strip()
# newline between star and wars and its at the end of the string:
tgtsegm = 'tomorrow I am planning on seeing star\nwars'

# allow for arbitrary white space between words:
split_words = topword.split()
regex_search = re.compile(rf'\b{re.escape(split_words[0])}\s+{re.escape(split_words[1])}\b')

if regex_search.search(srcsegm) and regex_search.search(tgtsegm):
    print('match in both')

Prints:

match in both