Dynamic Memory Allocation in fast RAM

Question

On a Windows 32-bit and 64-bit machine, I have to allocate memory to store large amounts of data that are streaming live, a total of around 1GB. If I use malloc(), I am going to obtain a virtual memory address, and this address could be actually causing some paging to the hard drive depending on the amount of memory I have. Unfortunately I'm afraid that the HD will impact performance and cause data to be missing.

Is there a way to force memory to allocate only in RAM, even if it means that I get an error when not enough memory is available (so the user needs to close other things or use another machine)? I want to guarantee that all operations will be done in memory. If this fails, forcing the application to exit is acceptable.

I know that another process may come in and itself take some memory, but I am not worried because in this machine that is not happening (it'll be the only application on the machine to be doing this large allocation).

[Edit:] My attempt so far has been to try use VirtualLock as follows:

if(!SetProcessWorkingSetSize(this, 300000, 300008))
    printf("Error Changing Working Set Size\n");

// Allocate 1GB space
unsigned long sz = sizeof(unsigned char)*1000000000;
unsigned char * m_buffer = (unsigned char *) malloc(sz);

if(m_buffer == NULL)
{
    printf("Memory Allocation failed\n");
}
else
{
    // Protect memory from being swapped
    if(!VirtualLock(m_buffer , sz))
    {
           printf("Memory swap protection failed\n");
    }           
}

But the change in Working set fails, and so does the VirtualLock. Malloc does return non-null.

[Edit2] I have tried also:

 unsigned long sz = sizeof(unsigned char)*1000000000;
 LPVOID lpvResult;
 lpvResult = VirtualAlloc(NULL,sz, MEM_PHYSICAL|MEM_RESERVE, PAGE_NOCACHE);

But lpvResult is 0, so no luck there either.

Answer 1

You can use mlock, mlockall, munlock, munlockall functions in order to prevent pages from being swapped (part of POSIX, also available in MinGW). Unfortunately, I have no experience with Windows but it looks like VirtualLock does the same thing.

Hope it helps. Good Luck!

Answer 2

I think VirtualAlloc might get you some of what you want.

This problem really boils down to just writing your own memory manager instead of using CRT function.

Answer 3

You need to use the undocumented NtLockVirtualMemory function with lock option 2 ( LOCK_VM_IN_RAM ); make sure you request and obtain SE_LOCK_MEMORY_NAME privilege first, and be aware that it might not be granted (I'm sure what the group policy defaults the privilege to, but it might very well be granted to nobody).

I suggest using VirtualLock as a fallback, and if that fails too, to use SetProcessWorkingSetSize . If that fails then just let it fail I guess...

See this link for some nice discussion about this. One person says:

When you specify LOCK_VM_IN_WSL flag, you just tell the Balance Set Manager that you don't want some particular page to get swapped to the disk, and ask it to leave this page alone when trimming the working set of the target process. This is just an indication, so that the target page may still get swapped if the system is low on RAM. However, when you specify LOCK_VM_IN_RAM flag, you issue a directive to the Memory Manager to treat this page as non-pageable (ie do something the driver does when it calls MmProbeAndLockPages () in order to lock pages, described by MDL) , so that the page is question is guaranteed to be loaded in RAM all the time.

---

Edit:

Read this .

Answer 4

One option would be to create a RAM Disk out of your host's memory. While there is no longer native support for this in the distributed Windows code, you can still find the drivers necessary for free or made available through commercial products. For instance, DRDataRam provides a free driver for personal use and a commercially licensed product for business use at: http://memory.dataram.com/products-and-services/software/ramdisk

There is also ImDisk Virtual Driver available at: http://www.ltr-data.se/opencode.html/#ImDisk It is open sourced and free for commercial use. It is digitally signed with a trusted certificate from Microsoft.

For more information concerning RAM Drives on Windows, check out ServerFault.com.

Answer 5

You should take a look at Address Windowing Extensions (AWE) . It sounds like it matches the memory constraints you have (emphasis mine):

AWE uses physical nonpaged memory and window views of various portions of this physical memory within a 32-bit virtual address space.