如何有效地定位句子中特定的单词序列

Question

The problem is to find a time efficient function that receives as inputs a sentence of words and a list of sequences of varying amounts of words (also known as ngrams) and returns for every sequence a list of indexes indicating where they occur in the sentence, and do it as efficiently as possible for large amounts of sequences.

What I ultimately want is to replace the occurrences of ngrams in the sentence for a concatenation of the words in the sequence by "_".

For example if my sequences are ["hello", "world"] and ["my", "problem"], and the sentence is "hello world this is my problem can you solve it please?" the function should return "hello_world this is my_problem can you solve it please?"

What I did is group the sequences by the amount of words each have and save that in a dictionary where the key is the amount and the value is a list of the sequences of that length.

The variable ngrams is this dictionary:

def replaceNgrams(line, ngrams):
    words = line.split()
    #Iterates backwards in the length of the sequences
    for n in list(ngrams.keys())[::-1]: #O(L*T)
        newWords = []
        if len(words) >= n:
            terms = ngrams[n]
            i = 0
            while i < len(words)+1-n: #O(L*Tn)
                #Gets a sequences of words from the sentences of the same length of the ngrams currently checking
                nwords = words[i:i+n].copy()
                #Checks if that sequence is in my list of sequences
                if nwords in terms: #O(Tn)
                    newWords.append("_".join(nwords))
                    i+=n
                else:
                    newWords.append(words[i])
                    i+=1
            newWords += words[i:].copy()
            words = newWords.copy()
    return " ".join(words)

This works as desired but I have too many sequences and too many lines to apply this too and this is way too slow for me (it would take a month to finish).

Answer 1

I think this can be achieved by basic string operations. I'll first join all the sequences into single strings and then look for them in the full_text . If found, I'll keep track of them in the output_dict with their start and end index. You can use these indices as you require.

full_text = "hello world this is my problem can you solve it please?"

sequences = [["hello", "world"], ["my", "problem"]]

joined_sequences = [" ".join(sequence) for sequence in sequences]

def find_location(message, seq):
    if seq in message:
        return message.find(seq)
    else:
        return None

output_dict = {}

for sequence in joined_sequences:
    start_index = find_location(full_text, sequence)
    if start_index > -1:
        output_dict[sequence] = [start_index, start_index+len(sequence)]

print(output_dict)

This will output:

{'hello world': [0, 11], 'my problem': [20, 30]}

Then you can do whatever you want with the start and end indices.

If you only need to replace the values with underscores in the middle, you might not even need the indices.

for sequence in joined_sequences:
    if sequence in full_text:
        full_text = full_text.replace(sequence, "_".join(sequence.split()))

print(full_text)

This should give you:

hello_world this is my_problem can you solve it please?