Allocate several GBs of memory for std::vector

Question

I need to acquire several GB of data from a sensor. When I tried to allocate a big array with malloc (10 or more GB. My system has 32GB) it returns NULL. So I thought the problem could be solved with a linked list of iterators to vectors.

However I don't know how to set this up. I tried declaring " list< vector::iterator >" but I can't allocate the memory for each vector (e/o should have 1000~2000 elements). Do you know any way to do this or maybe a better solution for this big memory allocation?

Answer 1

If you are using a 64-bit operating system, then malloc should be able to allocate the large size with no problem.

For example, this code runs on my windows machine (64-bit windows) and allocates 10GB of ram flawlessly:

#include <stdio.h>
#include <malloc.h>
int main(int argc, char *argv[]) {
    long int size = 10L * 1024 * 1024 * 1024;
    printf("size = %ld\n", size);
    char *x = (char *)malloc(size);
    printf("x = 0x%lx\n", x);
    long int i;
    for (i = 0; i < size; i += 1024*1024) {
        x[i] = 'h';
    }
    printf("Done1\n");
}

However, if you have a 32-bit operating system, you'll be in trouble, and can't allocate over some limit (maybe 3 GB, but probably system dependent)

In that case, you'll need to write your data to a file instead.

However, if you're using a fat filesystem, then you can't write to a file that big either. In that case, you'd have to split the data among many files under 2gb in size.

You'd want to actually check the malloc result for NULL to make sure the malloc works and memory could be grabbed.

Answer 2

I usually move to memory mapped files or shared memory maps for this kind of data volumes.

This way, you're not bound to the amount of physical (process) memory available at all. You can let the OS page in and out as required. Fragmentation becomes much less of an issue (unless you actually fragment the logical address space, which is quite hard to achieve on 64 bit architectures).

---

More information

I have quite a number of answers on SO that show examples of storing vectors and all manner of more complicated data structures in shared memory/mapped files. You might want to look for mapped_file_device (from Boost Iostreams) or managed_shared_memory and managed_mapped_file (from Boost Interprocess)

Answer 3

You will need to allocation this space under Windows 64 bit OS. You will ALSO have to set "large address space aware" flag, otherwise you can only get 2 GB of RAM due to how the virtual memory system works on Windows.

You may want to look into using a memory mapped file, as suggested by sehe in his answer if you do not absolutely have to have one large 10 GB chunk of continuous memory. If you have to build your application for Windows 32 bit, then this will be the only answer, as Windows 32 bit normally only allows for 2 GB of memory, unless the option is set for "large address space aware" flag, at which point it will allow 3 GB of memory usage.

Answer 4

当您必须处理大块内存时，最好完全跳过malloc并直接转到用于内存分配的操作系统调用。