有没有更快的方法来预处理 Python 中的大量文本数据？

Question

I'm building sentiment analyse algorithm to predict the score of IMDb reviews. I wanted to do it from scratch, so I scraped half a million reviews and created my own data set.

I'm sending small review packages (consist of 50 reviews) to review_cleaner with pool. It helped me to reduce run time from 40 minutes to 11 minutes for 1000 reviews. But, I have half a million reviews, I need faster way to process them. I was thinking if it's possible to run it on my GPU (GTX1060 6GB)? I installed CUDA, but I couldn't find how to run specific function(review_cleaner) on GPU cores.

Basically, what I need is, solution to run preprocess faster. I searched and tried many different things but couldn't do it. Is there any way to run it faster?

def filling_the_database(review_data): 
    try:
        c.executemany("""INSERT INTO preprocessed_reviews(review, review_score) VALUES (?, ?)""", review_data)
        conn.commit()
    except Error as e:
        print(e)


def get_wordnet_pos(word):
    """Map POS tag to first character lemmatize() accepts"""
    tag = nltk.pos_tag([word])[0][1][0].upper()
    tag_dict = {"J": wordnet.ADJ,
                "N": wordnet.NOUN,
                "V": wordnet.VERB,
                "R": wordnet.ADV}

    return tag_dict.get(tag, wordnet.NOUN)


def review_cleaner(review):
    lemmatizer = WordNetLemmatizer()
    new_review_data = ()
    bulk_data = []
    for each in review:
        review_temp = ''.join([i for i in each[0] if not i.isdigit()])
        review_temp = REPLACE_NO_SPACE.sub(" ", review_temp.lower())
        review_temp = nltk.word_tokenize(review_temp)
        review_temp = (lemmatizer.lemmatize(word, get_wordnet_pos(word)) for word in review_temp)
        review_temp = ' '.join([word for word in review_temp if word not in stopwords.words('english')])
        new_review_data = (review_temp, each[1])
        bulk_data.append(new_review_data)
    filling_the_database(bulk_data)


if __name__ == "__main__":
    review_data = ()
    bulk_data = []
    amount_of_reviews = 0
    previous_amount = 0
    conn = create_connection('2020-04-11')
    c = conn.cursor()
    c.execute("""CREATE TABLE IF NOT EXISTS preprocessed_reviews(review TEXT, review_score INTEGER, ID PRIMARY KEY)""")
    conn.commit()
    total_number_of_reviews = c.execute("""SELECT COUNT(*) FROM movie_reviews""")
    for each in total_number_of_reviews:
        total_number_of_reviews = each[0]
    while amount_of_reviews < total_number_of_reviews:
        review_package = []
        amount_of_reviews += 50
        data = c.execute("""SELECT * FROM movie_reviews WHERE ID BETWEEN (?) AND (?)""", (previous_amount, amount_of_reviews-1))
        conn.commit()
        previous_amount = amount_of_reviews
        for each in data:
            review_data = (each[0], each[1])
            review_package.append(review_data)
            del review_data
        bulk_data.append(review_package)
        del review_package
        print(amount_of_reviews)
    p = Pool(4)
    p.map(review_cleaner, bulk_data)
    p.close()
    print('---- %s seconds ----' % (time.time() - start_time))

I'm storing around half a million (400k) reviews in SQLite database. One column for review and one column for score of the review. In another table, I'm inserting the preprocessed reviews same way, one column for review and one column for score. I have 16 gigs of RAM, Intel i7 6700HQ, SSD and GTX1060 6GB.

Answer 1

A few thoughts crossed my mind.

1. Reading and writing from SQLite may have quite a lot of overhead when in reality you could fit 500k reviews in your 16GB of ram. You could do this by dumping your data to a tabulated csv file and then reading it in with pandas to do the preprocessing. You could also use pandarallel to parallelise the work instead of using pool to make your life easier.

2. If SQLite is not the bottleneck then it's likely a computational bottleneck in which case I would look at running the process overnight, or hiring a cloud-compute instance with good cpu resources. A 16core machine wouldn't be too expensive to rent on AWS for a short amount of time and that would give you a theoretical 4x speedup.