[英]How to limit memory usage within a python process
I run Python 2.7 on a Linux machine with 16GB Ram and 64 bit OS. 我在具有16GB Ram和64位操作系统的Linux机器上运行Python 2.7。 A python script I wrote can load too much data into memory, which slows the machine down to the point where I cannot even kill the process any more.
我写的一个python脚本可以将太多的数据加载到内存中,这会使机器速度降低到我甚至无法再杀死进程的程度。
While I can limit memory by calling: 虽然我可以通过调用来限制内存:
ulimit -v 12000000
in my shell before running the script, I'd like to include a limiting option in the script itself. 在运行脚本之前的shell中,我想在脚本本身中包含一个限制选项。 Everywhere I looked, the
resource
module is cited as having the same power as ulimit
. 在我看的每个地方,
resource
模块被引用为具有与ulimit
相同的功能。 But calling: 但是打电话:
import resource
_, hard = resource.getrlimit(resource.RLIMIT_DATA)
resource.setrlimit(resource.RLIMIT_DATA, (12000, hard))
at the beginning of my script does absolutely nothing. 在我的脚本开头做什么都没有。 Even setting the value as low as 12000 never crashed the process.
即使设置低至12000的值也从未使过程崩溃。 I tried the same with
RLIMIT_STACK
, as well with the same result. 我尝试使用
RLIMIT_STACK
,结果相同。 Curiously, calling: 奇怪的是,打电话:
import subprocess
subprocess.call('ulimit -v 12000', shell=True)
does nothing as well. 什么都不做。
What am I doing wrong? 我究竟做错了什么? I couldn't find any actual usage examples online.
我在网上找不到任何实际的用法示例。
edit: For anyone who is curious, using subprocess.call
doesn't work because it creates a (surprise, surprise!) new process, which is independent of the one the current python program runs in. 编辑:对于任何好奇的人,使用
subprocess.call
不起作用,因为它创建了一个(惊喜,惊喜!)新进程,它独立于当前python程序运行的进程。
resource.RLIMIT_VMEM
is the resource corresponding to ulimit -v
. resource.RLIMIT_VMEM
是与ulimit -v
对应的资源。
RLIMIT_DATA
only affects brk/sbrk
system calls while newer memory managers tend to use mmap
instead . RLIMIT_DATA
仅影响brk/sbrk
系统调用,而较新的内存管理器倾向于使用mmap
。
The second thing to note is that ulimit
/ setrlimit
only affects the current process and its future children. 需要注意的第二件事是
ulimit
/ setrlimit
只影响当前进程及其未来的子进程。
Regarding the AttributeError: 'module' object has no attribute 'RLIMIT_VMEM'
message: the resource
module docs mention this possibility: 关于
AttributeError: 'module' object has no attribute 'RLIMIT_VMEM'
消息: resource
模块docs提到了这种可能性:
This module does not attempt to mask platform differences — symbols not defined for a platform will not be available from this module on that platform.
此模块不会尝试屏蔽平台差异 - 在该平台上,此模块将无法使用未为平台定义的符号。
According to the bash
ulimit
source linked to above, it uses RLIMIT_AS
if RLIMIT_VMEM
is not defined. 根据
bash
ulimit
源连接到上述,它使用RLIMIT_AS
如果RLIMIT_VMEM
没有定义。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.