从词汇表替换字符串的有效方法-Python

Question

I have a vocabulary of phrases and I want to replace words of another file by these words. For example, I have the following vocabulary:

United States, New York

and I want to replace the following file:

"I work for New York but I don't even live at the United States"

To this:

"I work for New_York but I don't even live at the United_States"

Currently I'm doing at this way:

import os

def _check_files_and_write_phrases(docs, worker_num):
    print("worker ", worker_num," started!")
    for i, file in enumerate(docs):
        file_path = DOCS_FOLDER + file
        with open(file_path) as f:
            text = f.read()
            for phrase in phrases:
                text = text.replace(phrase, phrase.replace(' ','_'))
            new_doc = PHRASES_DOCS_FOLDER + file[:-4] + '_phrases.txt'
            with open(new_doc, 'w') as nf:
                nf.write(text)

    print("job done on worker ", worker_num)


docs = os.listdir(DOCS_FOLDER)

import threading

threads = []
for i in range(1, 11):
    print(i)
    start = int((len(docs)/10) * (i - 1))
    end = int((len(docs)/10) * (i))
    print(start,end)
    if i != 10:
        t = threading.Thread(target=_check_files_and_write_phrases, args=(docs[start:end], i, ))
    else:
        t = threading.Thread(target=_check_files_and_write_phrases, args=(docs[start:], i, ))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

print("all workers finished!")

But it's way too slow! I thought that threads would do the job but I was wrong...

Is there another efficient way of doing this?

Answer 1

All of the phrases could be replaced using a single re.sub() call which could be pre-compiled to further speed things up a bit:

import re

phrases = {"United States":"United_States", "New York":"New_York"}
re_replace = re.compile(r'\b({})\b'.format('|'.join(re.escape(phrase) for phrase in phrases.keys())))

def _check_files_and_write_phrases(docs, worker_num):
    print("worker {} started!".format(worker_num))

    for i, filename in enumerate(docs):
        file_path = DOCS_FOLDER + filename

        with open(file_path) as f:
            text = f.read()
            text = re_replace.sub(lambda x: phrases[x.group(1)], text)
            new_doc = PHRASES_DOCS_FOLDER + filename[:-4] + '_phrases.txt'

            with open(new_doc, 'w') as nf:
                nf.write(text)

    print("job done on worker ", worker_num)

This first creates a regular expression to search on as follows based on a dictionary of phrases:

\b(United\ States|New\ York)\b

The re.sub() function then uses the phrases dictionary to look up the required phrase replacement. It takes two parameters, the replacement and the original text. The replacement can either be a fixed string, or in this case a function is used. The function takes a single argument being the matched object, and returns the replacement text. A lambda function is used to do this, it simply looks up the match object in phrases dictionary.

Instead of doing a dictionary lookup, it could just use a replace() here but the pre-calculated replacement text should be faster. The \\b is added to only make replacements which are on word boundaries, so for example MYNew York would be skipped. Adding flags=re.I to the re.compile() could be used to make the search case insensitive if needed.

Answer 2

Try to change the for loop to replace only phrases that exist in the text:

for phrase in set(phrases).intersection(text.split()):
...

Try it with and without the threading.