[英]Why can a 352GB NumPy ndarray be used on an 8GB memory macOS computer?
import numpy as np
array = np.zeros((210000, 210000)) # default numpy.float64
array.nbytes
When I run the above code on my 8GB memory MacBook with macOS, no error occurs. 当我在装有macOS的8GB内存MacBook上运行上述代码时,没有发生错误。 But running the same code on a 16GB memory PC with Windows 10, or a 12GB memory Ubuntu laptop, or even on a 128GB memory Linux supercomputer, the Python interpreter will raise a MemoryError.
但是,如果在装有Windows 10的16GB内存PC,12GB内存Ubuntu笔记本电脑甚至是128GB内存Linux超级计算机上运行相同的代码,Python解释器将引发MemoryError。 All the test environments have 64-bit Python 3.6 or 3.7 installed.
所有测试环境都安装了64位Python 3.6或3.7。
@Martijn Pieters' answer is on the right track, but not quite right: this has nothing to do with memory compression, but instead it has to do with virtual memory . @Martijn Pieters的答案是正确的,但并非完全正确:这与内存压缩无关,而与虚拟内存有关 。
For example, try running the following code on your machine: 例如,尝试在计算机上运行以下代码:
arrays = [np.zeros((21000, 21000)) for _ in range(0, 10000)]
This code allocates 32TiB of memory, but you won't get an error (at least I didn't, on Linux). 这段代码分配了32TiB的内存,但是您不会收到错误消息(至少在Linux上没有)。 If I check htop, I see the following:
如果我检查htop,则会看到以下内容:
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
31362 user 20 0 32.1T 69216 12712 S 0.0 0.4 0:00.22 python
This because the OS is perfectly willing to overcommit on virtual memory . 这是因为OS完全愿意在虚拟内存上过量使用 。 It won't actually assign pages to physical memory until it needs to.
除非需要,否则它实际上不会将页面分配给物理内存。 The way it works is:
它的工作方式是:
calloc
asks the OS for some memory to use calloc
要求操作系统使用一些内存 Creating a single huge array doesn't work on Linux because, by default, a "heuristic algorithm is applied to figure out if enough memory is available". 创建单个大数组在Linux上不起作用,因为默认情况下, “采用启发式算法来确定是否有足够的内存”。 ( thanks @Martijn Pieters! ) Some experiments on my system show that for me, the kernel is unwilling to provide more than
0x3BAFFFFFF
bytes. ( 感谢@Martijn Pieters! )对我的系统进行的一些实验表明,对于我来说,内核不愿意提供超过
0x3BAFFFFFF
字节。 However, if I run echo 1 | sudo tee /proc/sys/vm/overcommit_memory
但是,如果我运行
echo 1 | sudo tee /proc/sys/vm/overcommit_memory
echo 1 | sudo tee /proc/sys/vm/overcommit_memory
, and then try the program in the OP again, it works fine. echo 1 | sudo tee /proc/sys/vm/overcommit_memory
,然后再次在OP中尝试该程序,它可以正常工作。
For fun, try running arrays = [np.ones((21000, 21000)) for _ in range(0, 10000)]
. 为了好玩,尝试运行
arrays = [np.ones((21000, 21000)) for _ in range(0, 10000)]
。 You'll definitely get an out of memory error, even on MacOs or Linux with swap compression. 即使在MacO或具有交换压缩功能的Linux上,您也绝对会遇到内存不足的错误。 Yes, certain OSes can compress RAM, but they can't compress it to the level that you wouldn't run out of memory.
是的,某些操作系统可以压缩RAM,但无法将其压缩到不会耗尽内存的水平。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.