如何在Windows上保留内存并稍后将文件映射到其中？

Question

I would like to reserve a region of memory and later map files contiguously into the reserved memory. There may be large time gaps between mapping the files during which other functions may allocate memory from the heap. Once mapped, a file may not be unmapped and mapped to a new memory location.

On Linux that would be something like:

#include <iostream>
#include <sys/mman.h>
#include <unistd.h>
#include <fcntl.h>
#include <cerrno>

int main(){
    void *memory = mmap(nullptr, getpagesize() * 2,
                        PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE | MAP_NORESERVE, -1, 0); // reserve memory
    int handle1 = ::open("1", O_CREAT | O_RDWR, S_IRWXU); // open file1
    int handle2 = ::open("2", O_CREAT | O_RDWR, S_IRWXU); // open file2
    void *data = mmap(memory, getpagesize(), PROT_READ | PROT_WRITE, MAP_SHARED |MAP_FIXED, handle1, 0); // map first file into reserved memory
    void *data2 = mmap(static_cast<char *>(memory) + getpagesize(), getpagesize(), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, handle2, 0); // map second file into reserved memory
}

On Windows however I can't seem to find the correct way to do this. Does anyone know how this is done?

Edit: Since the my goal seems to be not that easy to understand. Once again simplified:

I would like to have memory that is memory mapped. Once the mapped memory is filled with data I would like to map a new file into memory directly after the already mapped memory to extend the mapped memory (without leaving a gap between the two memory mapped regions).

At a later program start the files may be used to restore all the data of the previous run.

Problem with solution with sparse files and extending existing file: Program should be able to remove files if they aren't needed any more. Therefore its important that the mappings are always new files.

You may think of it more as an allocator. Memory mapped memory is needed. The library maps a block of memory and returns a pointer to a subblock. The memory is no longer needed, it is returned to the allocator. In case an entire mapping is no longer needed, the associated file is removed (The data does not need to be written by the mapping if it is no longer needed).

Answer 1

You don't really need to have it reserved when you map in the files, you just need to know that you can map in the two files contiguously somewhere. For example your Linux example code, being single threaded, would work just as well if you immediately unmapped the reserved region before mapping in the files.

On Windows, handling the possible multithreaded race condition, you can do something like:

 while(1) {
     char *memory = VirtualAlloc(NULL, page_size * 2, MEM_RESERVE, 0);
     VirtualFree(memory);
     if (MapViewOfFileEx(handle1, FILE_MAP_WRITE, 0, 0, page_size, memory) == NULL
         && GetLastError() == ERROR_INVALID_ADDRESS) {
         continue;
     }
     if (MapViewOfFileEx(handle2, FILE_MAP_WRITE, 0, 0, page_size, memory + page_size) == NULL
         && GetLastError() == ERROR_INVALID_ADDRESS) {
         UnMapViewOfFile(memory);
         continue;
     }
     break;
}

Answer 2

my solution used UNDOCUMENTED api

NTSYSAPI NTSTATUS NTAPI ZwExtendSection ( HANDLE SectionHaqndle, PLARGE_INTEGER SectionSize );

no win32 analog for this function, but this is key point of solution. also we need use ZwMapViewOfSection but not MapViewOfFileEx (win32 shell over ZwMapViewOfSection ) because MapViewOfFileEx have less parameters than ZwMapViewOfSection - we cannot set ULONG AllocationType to MEM_RESERVE - but this is also key point. for other tasks we can use win32 analog, but for uniformity and style i will be use NT api.

of course many just say that it undocumented, unsupported, etc use ntdll api direct - but really it work and here i not view solution based on win32 only. so who want can use, who not want can not use. as is

the idea - we just reserve required large region with call to ZwMapViewOfSection (this is impossible done by MapViewOfFileEx ) and then, when need we can extent this region by call to ZwExtendSection

solution tested and worked.

class SECTION_EX
{
    LARGE_INTEGER _CurrentSize, _MaximumSize;
    HANDLE _hSection;
    PVOID _BaseAdress;

public:
    NTSTATUS Create(POBJECT_ATTRIBUTES poa, SIZE_T InitialSize, SIZE_T MaximumSize);
    NTSTATUS Extend(SIZE_T NewSize);

    SECTION_EX()
    {
        _BaseAdress = 0;
        _hSection = 0;
    }

    ~SECTION_EX()
    {
        if (_hSection) 
        {
            if (_BaseAdress) ZwUnmapViewOfSection(NtCurrentProcess(), _BaseAdress);
            ZwClose(_hSection);
        }
    }
};

NTSTATUS SECTION_EX::Extend(SIZE_T NewSize)
{
    LARGE_INTEGER Size;
    Size.QuadPart = NewSize; 

    if (Size.QuadPart <= _CurrentSize.QuadPart)
    {
        return STATUS_SUCCESS;
    }

    if (Size.QuadPart > _MaximumSize.QuadPart)
    {
        return STATUS_SECTION_TOO_BIG;
    }

    NTSTATUS status = ZwExtendSection(_hSection, &Size);

    if (0 <= status)
    {
        _CurrentSize = Size;
    }

    return status;
}

NTSTATUS SECTION_EX::Create(POBJECT_ATTRIBUTES poa, SIZE_T InitialSize, SIZE_T MaximumSize)
{
    HANDLE hFile;
    IO_STATUS_BLOCK iosb;

    NTSTATUS status = ZwCreateFile(&hFile, FILE_GENERIC_READ|FILE_GENERIC_WRITE, poa,
        &iosb, 0, 0, FILE_SHARE_VALID_FLAGS, FILE_OPEN_IF, FILE_SYNCHRONOUS_IO_ALERT, 0, 0);

    if (0 <= status)
    {
        _MaximumSize.QuadPart = MaximumSize;

        LARGE_INTEGER Size, *pSize = &Size;
        Size.QuadPart = InitialSize;

        if (iosb.Information == FILE_OPENED)
        {
            FILE_STANDARD_INFORMATION fsi;

            if (0 <= (status = ZwQueryInformationFile(hFile, &iosb, &fsi, sizeof(fsi), FileStandardInformation)))
            {
                if (fsi.EndOfFile.QuadPart)
                {
                    pSize = 0;// in case file already exist with not zero size - use it
                }
            }
        }

        if (0 <= status)
        {
            status = ZwCreateSection(&_hSection, SECTION_ALL_ACCESS, 0, pSize, 
                PAGE_READWRITE, SEC_COMMIT, hFile);
        }

        ZwClose(hFile);

        if (0 <= status)
        {
            SECTION_BASIC_INFORMATION sbi;

            if (0 <= ZwQuerySection(_hSection, SectionBasicInformation, &sbi, sizeof(sbi), 0))
            {
                _CurrentSize = sbi.Size;// real file size in bytes, without align

                // !!! use MEM_RESERVE !!!
                // MaximumSize - will be reserved, but not all commited

                status = ZwMapViewOfSection(_hSection, NtCurrentProcess(), &_BaseAdress, 0, 
                    0, 0, &MaximumSize, ViewUnmap, MEM_RESERVE, PAGE_READWRITE);
            }
        }
    }

    return status;
}

void demoS()
{
    SECTION_EX se;
    STATIC_OBJECT_ATTRIBUTES(oa, "\\??\\c:\\***");
    // reserve 256Mb,but initially commit only 32kb or file size
    if (0 <= se.Create(&oa, 0x8000, 0x10000000))
    {
        se.Extend(0x18000);
        se.Extend(0x1e245);
        se.Extend(0x74100);
    }
}

---

update: i discover that begin from win 8.1 we can reserve memory region with section by used also undocumented FILE_MAP_RESERVE - so need call

_BaseAdress = MapViewOfFileEx(_hSection, FILE_MAP_ALL_ACCESS|FILE_MAP_RESERVE, 0, 0, MaximumSize, 0);

but in windows 7 , vista and XP - this will be not work. however ZwMapViewOfSection worked with MEM_RESERVE flag even in XP.

so usual situation - frequently win32 shell have less functional compare corresponded Nt*/Zw* function.

and still no any win32 analog/shell for ZwExtendSection (this call extend both file and view)

Answer 3

The correct solution is to rearchitecture to eliminate the requirement for adjacent mappings.

Depending on your exact needs, one approach would be to use a single sparse file ( as described here ) whose initial length is equal to the amount of address space you want to reserve. Because the file is sparse, only the blocks that are actually used take up space on disk.

Failing that, you may need to change the underlying algorithms that are processing the data so that they no longer depend on the memory being contiguous. This usually isn't as hard as it may sound.

---

One (not at all proper!) alternative would be to hook the VirtualAlloc function so that you can make it block as necessary. That would allow you to manipulate your reserved memory range in an effectively atomic manner - you can free it, map part of it, and then re-reserve the rest, similar to Ross Ridge's answer, without any concern that another thread will allocate the memory while you are doing so.

(That wouldn't protect you from device drivers, but AFAIK it is extremely rare for a device driver to spontaneously allocate memory in user address space.)

NB: I'm assuming here that using a single file isn't acceptable for some reason. If a single file will do, you should use a sparse file as suggested above. If for some reason a single file will do but using a sparse file isn't an option, I would recommend RbMm's approach over this one - neither is a good solution, but my best guess is that this one is marginally more risky. (It is certainly more difficult.)