在pandas df中预处理大量文本的更有效方法？

Question

I have a series of text preprocessing steps organized in the following text_preprocessing() function below. (There are some more to it, such as converting emojis, removing punctuations etc., I dropped those for clarity.)

import spacy
nlp_model = spacy.load('en_core_web_lg')
nlp_model.add_pipe("merge_entities")

def text_preprocessing(text, lemmatizer):

    text = text.lower()
    text = " ".join([lemmatizer.lemmatize(w) for w in text.split()])
    text = [w if not re.search(r'[^\x00-\x7F]', w) else "<FOREIGN>" for w in text.split()]
    text = [w.text if (not w.ent_type_ or w.ent_type_ == 'PERSON' or w.ent_type_ == 'ORG')
            else f"<{w.ent_type_}>" for w in nlp_model(" ".join(text))]
    text = " ".join(text)
    text = re.sub(r"<\s([A-Z]+?)", r"<\1", text)
    text = re.sub(r"([A-Z]+?)\s>", r"\1>", text)
    text = re.sub(r"\s(<[A-Z]+?)\s", r" \1> ", text)
    text = re.sub(r"\s([A-Z]+?>)\s", r" <\1 ", text)
    text = " ".join([w.upper() if ("<" in w and ">" in w) else w for w in text.split()])
    return text

At the moment, I have a working solution which is as follows:

from nltk.stem import WordNetLemmatizer
lmtzr = WordNetLemmatizer()
df['Preprocessed'] = df['Text'].apply(lambda x: text_preprocessing(x, lmtzr))

I already moved the instantiation of WordNetLemmatizer outside of text_preprocessing() and passed the instance as an argument. Now I am thinking of further optimizing this code as the database of messages to run on has increased considerably and is now nearing 30,000 rows (30,000 texts to preprocess, and the amount is growing day-by-day). Text preprocessing one-by-one takes plenty of hours already. I tried multiprocessing.Process earlier but didn't make much of an impact. I read about vectorization but I'm unsure how it could be applied to my situation. I'm also aware of external packages that apparently make it easier to set up multiprocessing for df.apply() , such as the swifter module, but I am hoping to speed things up a bit more than 2-4 times since I already have quite a lot of data and this will be even more in the future.

Example data can be created with the following function:

import pandas as pd
import numpy as np
import random
def generate_example_data(rows=100):
    df = pd.DataFrame(np.random.randint(0,100,size=(rows, 4)), columns=list('ABCD'))
    df['Text'] = pd.Series(["".join([random.choice("aábcčdeëfghijklmnoópqrsştuvwxyz    ") for i in range(random.randint(25,400))]) for j in range(rows)])
    return df

Answer 1

I think, your reading re vector calculation is the key here and I would go that way before considering multithreading or multiprocessing. There are some operations that can be vectorized on your DataFrame already. You shouldnt care too much about adding columns to your frame.

for example your first operation

text = text.lower()

can be replaced with

df['some_col'].str.lower()

Furthermore, you can substitute some of your regex operations using this thread. Also, i find this a good source. In addition, try (not too sure your case is a good fit) as much as you can to make use of numpy library.

good luck