处理内存中的大量文本数据

Question

I am trying to process ~20GB of data on a Ubuntu system having 64 GB of RAM.

This step is a part of a some preprocessing steps to generate feature vectors for training an ML algo.

The original implementation(written by someone in my team) had lists in it. It does not scale up well as we add more training data. It is something like this.

all_files = glob("./Data/*.*")
file_ls = []

for fi in tqdm(all_files):
    with open(file=fi, mode="r", encoding='utf-8', errors='ignore') as f:
        file_ls.append(f.read())

This runs into a memory error(process gets killed). So I though I should try out replacing the list based thing with tries

def insert(word):
    cur_node = trie_root
    for letter in word:
        if letter in cur_node:
            cur_node = cur_node[letter] 
        else:
            cur_node[letter] = {} 
            cur_node = cur_node[letter]
    cur_node[None] = None

trie_root = {}

for fi in tqdm(all_files):
    with open(file=fi, mode="r", encoding='utf-8', errors='ignore') as f:
        insert(f.read().split())

This too gets killed. The above is a demo code that I have written to capture the memory footprint of the objects. The worse part is that the demo code for list runs standalone but the demo code for trie gets killed, leading me to believe that this implementation is worse than the list implementation.

My goal is to write some efficient code in Python to resolve this issue.

Kindly help me solve this problem.

EDIT: Responding to @Paul Hankin, the data processing involves first taking up each file and adding a generic placeholder for terms with a normalized term frequency greater than 0.01 after which each file is splitted into a list and a vocabulary is calculated taking all the processed files into consideration.

Answer 1

One of the simple solutions to this problem might be to NOT store data in a list or any data structure. You can try writing these data to a file while doing the reading.