如何在不占用大量内存的情况下读取泡菜文件

Question

Currently I have a loop that contains:

df = pandas.read_pickle(filepath)

the files are ~ 100 mb. However since it is looping through these many times, it is taking up a lot of memory and then eventually I get a memory error. Is there a way to do this where I can close the file once I get the dataframe?

My code is correct because I've tested it with less files. I just want an alternative to reading large pickle files without taking up too much memory after I read it.

Answer 1

del df
gc.collect()

Erase reference, and garbage colector.

Edit, this erase your dataframe from memory, you can not close the file, take the info and dont use ram.