[英]Why do I get heap outOfMemory exception?
I'm trying to allocate a large matrix (around 10GB). 我正在尝试分配一个大矩阵(大约10GB)。 I'm working on 64 bit machine with a 64 bit JVM. 我正在使用64位JVM的64位机器。 My process then should have 2^64
bytes available and I've set the JVM heap size to be 128G (I have 16GB of RAM in my machine if that matters). 然后我的进程应该有2^64
字节可用,并且我已经将JVM堆大小设置为128G(如果重要的话,我的机器中有16GB的RAM)。 My understanding was that I should get the memory from the OS and that unneeded matrix cells would be swapped out by the OS. 我的理解是我应该从操作系统获取内存,操作系统会将不需要的矩阵单元换出。 However I'm getting the above exception. 但是我得到了上述异常。
Edit: 编辑:
This is how I'm defining the matrix: 这就是我定义矩阵的方式:
Jama.Matrix A = new Matrix(num_words, num_documents);
Where num_words
is approximately 100k and num_documents
is approximately 35k. 其中num_words
约为100k, num_documents
约为35k。 Also worth mentioning that the type is double
另外值得一提的是,这种类型是double
Edit2: EDIT2:
Relevant flags: 相关标志:
-Xms40m
-Xmx128g
-d64
JVM works as native process, in this respect: JVM requests memory from OS that can allocate it either in RAM in swap. 在这方面,JVM作为本机进程工作:JVM从OS请求内存,可以在交换机的RAM中分配它。
The memory you can allocate in java does not depend on your RAM but on command line option -Xmx
you specify when you are running your JVM. 您可以在java中分配的内存不依赖于您的RAM,而是依赖于您在运行JVM时指定的命令行选项-Xmx
。 If it is not enough memory in RAM JVM receives it from swap and (I believe) even does not know about that. 如果RAM内存不足,JVM会从交换中接收它(我相信)甚至不知道这一点。
However, 然而,
If you need to work with large data you need to work with BigMemory products (EhCache or Terracotta). 如果您需要处理大数据,则需要使用BigMemory产品(EhCache或Terracotta)。
Finally, run jvisualvm or with the -verbose:gc
prameter to see the heap allocation. 最后,运行jvisualvm或使用-verbose:gc
查看堆分配。
Here some description: 这里有一些描述:
Xms -> the init memory that should be allocated in the start up in MB.
Xmx -> the Max amount of memory that your application can get in MB e.g Xmx2048m.
-XX:MaxNewSize= -> the max S1 and S2 memory size
-XX:NewSize= -> init S1 and S2 size
so in your case if you want to allocate memory as much as you can, so you need to say for example 16 * 1024 = 16384 or 16g
and the -XX:MaxNewSize= and -XX:NewSize=
set it by 40% of your Xmx 所以在你的情况下,如果你想尽可能多地分配内存,那么你需要说例如16 * 1024 = 16384 or 16g
和-XX:MaxNewSize= and -XX:NewSize=
将它设置为你的40% XMX
A few things you should know. 你应该知道一些事情。
-d64
only works on Solaris. -d64
仅适用于Solaris。 In short you need about 256 GB of main memory and a large heap you really consider what you are doing. 简而言之,您需要大约256 GB的主内存和大堆,您真正考虑的是您在做什么。 Accessing memory randomly is up to one million times faster than accessing disk space and this means not only is it one million times slower, but it is unlikely to work at all. 随机访问内存比访问磁盘空间快一百万倍,这意味着它不仅速度慢一百万倍,而且根本不可能工作。
What you can do is use off heap memory, and if you really know what you are doing, you can make this work and be perhaps only 100x slower than having the memory you need. 你可以做的就是使用堆内存,如果你真的知道你在做什么,你可以使它工作,并且可能只比你需要的内存慢100倍。 esp if you use a fast SSD for swap or your matrix is sparse is just the right ways. esp如果你使用快速SSD进行交换,或者你的矩阵是稀疏的,那就是正确的方法。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.