How to allocate a single array that does not fit in RAM

Question

I thought I'd never say this. I would like to make my machine run super-slow.

Here's the manner in which I'd like to do this: I'd like to allocate a single large array in F# (Array.init). It should be so large that random accesses into the array should generate page faults. I have 4GB RAM, and running in 64-bit mode. When I allocate 2^29 4-byte integers, runtime throws an out of memory exception.

> Array.init (1 <<< 28) (fun i->i);;
val it : int [] =
  [|0; 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20;
    21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39;
    40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58;
    59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77;
    78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96;
    97; 98; 99; ...|]
> Array.init (1 <<< 29) (fun i->i);;
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at <StartupCode$FSI_0003>.$FSI_0003.main@()
Stopped due to error
> 

Answer 1

The maximum size allowed by the Microsoft CLR for a single object is 2GB, regardless of whether you're running on a 32-bit or 64-bit platform.

You're running up against this hard limit when you try to allocate the array of 2**29 integers. (The array's data would be exactly 2GB, but objects also need a few extra bytes for housekeeping etc, pushing you over 2GB.)

Try allocating a slightly smaller array to allow for those extra few bytes of overhead. I can't remember exactly how much smaller it'd need to be -- a few experiments with (2**29)-3 integers, (2**29)-4 integers, (2**29)-5 integers etc should tell you pretty quickly.