How and when is the memory of a global or static array allocated?

Question

When defining a global or static array in c++ its memory is not immediately reserved at the start of the programme but only once we write to the array. What I found surprising is, if we only write to a small part of the array it still does not reserve the entire memory. Consider the following small example which writes sparsely to the global array:

#include <cstdio>
#include <cstdlib>

#define MAX_SIZE 250000000
double global[MAX_SIZE];

int main(int argc, char** argv) {
   if(argc<2) {
      printf("usage: %s <step size>\n", argv[0]);
      exit(EXIT_FAILURE);
   }
   size_t   step_size=atoi(argv[1]);

   for(size_t i=0; i<MAX_SIZE; i+=step_size) {
      global[i]=(double) i;
   }

   printf("finished\n"); getchar();
   return EXIT_SUCCESS;
}

Now executing this for different step sizes and looking at the output of top we get for example:

./a.out 1000000
./a.out 100000
./a.out 10000
./a.out 1000
./a.out 100

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15718 user      20   0 1918m 1868  728 S    0  0.0   0:00.00 a.out
15748 user      20   0 1918m  10m  728 S    0  0.1   0:00.00 a.out
15749 user      20   0 1918m  98m  728 S    1  0.8   0:00.04 a.out
15750 user      20   0 1918m 977m  728 S    0  8.1   0:00.39 a.out
15751 user      20   0 1918m 1.9g  728 S   23 15.9   0:00.80 a.out

The RES column indicates memory is only reserved in small blocks, which also means the array is unlikely to be contiguous in physical memory. Anyone got more insight on the lower level of things?

This also has the negative side effect that I can easily run many programmes where the sum of all VIRT exceeds the physical memory as long the sum of RES is below. However, as soon as they all write to the global arrays the system runs out of physical memory and programmes get sent sigkill or something.

Ideally I'd like to tell the compiler to reserve the memory of global and static variables at the start. Possible?

Edit

@Magnus: The lines are actually in the right order. :) Take the first line for example ./a.out 1000000 means I'm writing every 1 millionth entry in the array and so only 250 in total. This corresponds to a RES of only 1868k. In the last example ./a.out 100 every one hundreds entry is written and then the total memory is also physically allocated RES=VIRT=1.9g. From the numbers it appears that whenever an entry is written to the array something like a full 4k block is reserved on the physical memory.

@Nawaz: The array is contiguous in virtual address space but as I understand the OS might be lazy and only reserve physical memory when it's actually needed. Since this is done in little blocks and not the whole array at once how can it be guaranteed to be contiguous in physical memory?

@Nemo: Thanks for that, indeed when calling multiple instances of a.out which pause at the beginning and then write to the array I got oom-killer messages in /var/log/messages , and indeed your sysctrl command prevents me from starting too many a.out instances in the first place. Thanks!

Jun  1 17:49:16 localhost kernel: [32590.293421] a.out invoked oom-killer: gfp_mask=0x280da, order=0, oomkilladj=0
Jun  1 17:49:18 localhost kernel: [32592.110033] kded4 invoked oom-killer: gfp_mask=0x201da, order=0, oomkilladj=0
Jun  1 17:49:20 localhost kernel: [32594.718757] firefox invoked oom-killer: gfp_mask=0x201da, order=0, oomkilladj=0

The last two lines slightly worrying. :)

@doron: Thanks, great explanation, sorry can't upvote/select.

Answer 1

You are looking at virtual memory pages being committed. The OS will typically only commit pages when they are expressly written or read by your code. This has nothing to do with C++, which guarantees that arrays are contiguous. If you are asking how to get your OS to commit all your process's pages at start up, you need to use OS specific stuff (if such exists).

Answer 2

There are two things in play here viz. virtual memory and physical memory.

The virtual memory for for the static data, just like the instructions for your program are assigned before your program begins execution. By this I mean that the address, for your program is always defined.

The operating system might be lazy, however when it comes to loading both static data and program instructions into physical memory RAM. The way that this works is like this:

• The process loader assigned process virtual memory for the static data but does not load the data into RAM.
• When one tries to access these addresses, a processor exception is triggered and we then enter kernel mode.
• The kernel now loads the data into RAM and links the RAM into the processes virtual address space.
• The kernel switches back into user mode to the exact point where the processor exception happenned.
• Since the RAM is now linked into the processes virtual address space, the program will now continue executing as if nothing had ever happenned.

This is a total slight of hand which the operating system is allowed to do simply because it is totally undetectable to the running process. Unless of course we are short of memory.

Answer 3

I don't think the table which you posted proves anything substantial.

As far as, array of static storage is concerned, its allocated before the start of the program, which by definition means, before the program enters into the main() function, the runtime allocates memory to the global arrays, and it lasts for duration of the program:

§3.7.1/1

All objects which neither have dynamic storage duration nor are local have static storage duration. The storage for these objects shall last for the duration of the program (3.6.2, 3.6.3).

And whether it is global or local, arrays have always contiguous memory.

Answer 4

This sounds like a Linux system, where the "OOM killer" wakes up and starts killing processes once the used memory exceeds the available virtual memory. Grep for "oom" in /var/log/messages to confirm.

If so, this setting:

sysctl -w vm.overcommit_memory=2

...will prevent the kernel from allowing your processes to allocate more than the available VM.