[英]Memory leak when reading data file
I wrote a script to import data from a (quite messy) datafile. 我编写了一个脚本来从(相当混乱的)数据文件中导入数据。 Each line is read and processed separately in a loop.
循环读取和处理每行。
I wrote the following code to skip the header and white lines: 我编写了以下代码以跳过标题和空白行:
for line in rd_file.readlines():
line_1 = line.rstrip("\n")
# Decide what to do based on the content in the line.
if "#" in line.lower():
header_flag=True
# Don't print the header
pass
elif line.strip() == "":
pass
else:
[...]
Running the script I noticed a memory leak. 运行脚本时,我发现内存泄漏。 I located it using memory_profiler and I found out it is due to:
我使用memory_profiler找到了它,我发现这是由于:
elif line.strip() == "":
pass
This is what I get from memory_profiler
: 这是我从
memory_profiler
得到的:
45 204.5 MiB 160.6 MiB elif line.strip() == ""
How is it possible that 160 MB get occupied just by skipping a blank line? 仅跳过空白行,怎么可能占用160 MB? Do you have any suggestion on how to fix this?
您对如何解决此问题有任何建议吗?
I recommend not invoking readlines(), but instead depend on the python file iterator pattern. 我建议不要调用readlines(),而要依赖python文件迭代器模式。
for line in rd_file:
line_1 = line.rstrip("\n")
...
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.