如何通过知道 python 中单词的偏移量从文本文件中获取原始句子？

Question

I am new to python and I wonder if there is an efficient way to find the original sentence from a text file by knowing an offset of a word. Suppose that I have a test.txt file like this:

test.txt

Ceci est une wheat phrase corn.
Ceci est une deuxième phrase barley.
This is the third wheat word.

Suppose that I know the offset of the word "wheat" which is [13,18].

My codes look like this:

import nltk
from nltk.tokenize import word_tokenize

with open("test.txt") as f:
    list_phrase = f.readlines()
    f.seek(0)
    contents = f.read()
    for index, phrase in enumerate(list_phrase):
        j = word_tokenize(phrase)
        if contents[13:18] in j:
            print(list_phrase[index])

The output of my codes will print both sentences ie ( "Ceci est une wheat phrase corn." and "This is the third wheat word." )

How to detect exactly the real phrase of a word by knowing its offset?

Note that the offset that I considered continues between many sentences (2 sentences in this case). For example, the offset of the word "barley" should be [61,67].

The desire output of the print above should be:

Ceci est une wheat phrase corn.

As we know that its offset is [13,18].

Any help for this would be much appreciated. Thank you so much!

Answer 1

If you are looking for raw speed then the standard library is probably the best approach to take.

# Generate a large text file with 10,000,001 lines.
with open('very-big.txt', 'w') as file:
    for _ in range(10000000):
        file.write("All work and no play makes Jack a dull boy.\n")
    file.write("Finally we get to the line containing the word 'wheat'.\n")

Given the search_word and its offset in the line we're looking for we can calculate the limit for the string comparison.

search_word = 'wheat'
offset = 48
limit = offset + len(search_word)

The simplest approach is to iterate over the enumerated lines of text and perform a string comparison on each line.

with open('very-big.txt', 'r') as file:
    for line, text in enumerate(file, start=1):
        if (text[offset:limit] == search_word):
            print(f'Line {line}: "{text.strip()}"')

The runtime for this solution is 155 ms on a 2012 Mac mini (2.3GHz i7 CPU). That seems pretty fast for processing 10,000,001 lines but it can be improved upon by checking the length of the text before attempting the string comparison.

with open('very-big.txt', 'r') as file:
    for line, text in enumerate(file, start=1):
        if (len(text) >= limit) and (text[offset:limit] == search_word):
            print(f'Line {line}: "{text.strip()}"')

The runtime for the improved solution is 71 ms on the same computer. It's a significant improvement but of course mileage will vary depending on the text file.

Generated output:

Line 10000001: "Finally we get to the line containing the word 'wheat'."

EDIT: Including file offset information

with open('very-big.txt', 'r') as file:
    file_offset = 0
    for line, text in enumerate(file, start=1):
        line_length = len(text)
        if line_length >= limit and (text[offset:limit] == search_word):
            print(f'[{file_offset + offset}, {file_offset + limit}] Line {line}: "{text.strip()}"')
        file_offset += line_length

Sample output:

[430000048, 430000053] Line 10000001: "Finally we get to the line containing the word 'wheat'."

Encore une fois

This code checks if the known offset of the text is between the values of the offset of the start of the current line and the end of the line. The text found at the offset is also verified.

long_string = """Ceci est une wheat phrase corn.
Ceci est une deuxième phrase barley.
This is the third wheat word.
"""

import io

search_word = 'barley'
known_offset = 61
limit = known_offset + len(search_word)

# Use the multi-line string defined above as file input
with io.StringIO(long_string) as file:
    file_offset = 0
    for line, text in enumerate(file, start=1):
        line_length = len(text)
        if file_offset < known_offset < (file_offset + line_length) \
        and (text[(known_offset-file_offset):(limit-file_offset)] == search_word):
            print(f'[{known_offset},{limit}]\nLine: {line}\n{text}')
        file_offset += line_length

Output:

[61,67]
Line: 2
Ceci est une deuxième phrase barley.

Answer 2

If you already know the position of the word, tokenizing is not what you want to do. By tokenizing, you change the sequence (for which you know the position) to a list of words, where you don't know which element is your word.

Therefore, you should leave it at the phrase and just compare the part of the phrase with your word:

with open("test.txt") as f:
    list_phrase = f.readlines()
    f.seek(0)
    contents = f.read()
    for index, phrase in enumerate(list_phrase):
        if phrase[13:18].lower() == "wheat": ## .lower() is only necessary if the word might be in upper case.
            print(list_phrase[index])

This would only return the sentences where wheat is at the position [13:18] . All other occurrences of wheat would not be recognized.