如何防止 multiprocessing.pool 消耗我的所有内存？

Question

My multiprocessing pool (8 cores, 16 GB RAM) is using all of my memory before ingesting much data. I am operating on a 6 GB dataset

I have tried using various types of processors, including imap, imap_unordered, apply, map, etc. I have also tried maxtasksperchild, which seems to increase memory usage.

import string
import re
import multiprocessing as mp
from tqdm import tqdm

linkregex = re.compile(r"http\S+")
puncregex = re.compile(r"(?<=\w)[^\s\w](?![^\s\w])")
emojiregex = re.compile(r"(\u00a9|\u00ae|[\u2000-\u3300]|\ud83c[\ud000-\udfff]|\ud83d[\ud000-\udfff]|\ud83e[\ud000-\udfff])")

sentences = []

def process(item):
    return re.sub(emojiregex, r" \1 ", re.sub(puncregex,"",re.sub(linkregex, "link", item))).lower().split()


if __name__ == '__main__':
    with mp.Pool(8) as pool:
        sentences = list(tqdm(pool.imap_unordered(process, open('scrape/output.txt')
), total=52123146))

print(str(len(sentences)))
with open("final/word2vectweets.txt", "a+") as out:
    out.write(sentences)

This should return a list of processed lines from the file, but it consumes memory too fast. Previous versions with no mp and simpler processing have been successful.

Answer 1

How does this look?

import re
import multiprocessing as mp

linkregex = re.compile(r"http\S+")
puncregex = re.compile(r"(?<=\w)[^\s\w](?![^\s\w])")
emojiregex = re.compile(r"(\u00a9|\u00ae|[\u2000-\u3300]|\ud83c[\ud000-\udfff]|\ud83d[\ud000-\udfff]|\ud83e[\ud000-\udfff])")


def process(item):
    return re.sub(emojiregex, r" \1 ", re.sub(puncregex,"",re.sub(linkregex, "link", item))).lower().split()


with mp.Pool() as pool, open(in_file_path, 'r') as file_in, open(out_file_path, 'a') as file_out:
    for curr_sentence in pool.imap_unordered(process_line, file_in, chunksize=1000):
        file_out.write(f'{curr_sentence}\n')

I tested a bunch of chunk sizes, 1000 seems to be the sweet spot. I will keep investigating.