使用mmap时性能下降

Question

I have to compute a huge nxn matrix (n > 100000) and somehow store it in memory for further usage. Computation of a single element is quite expensive ( a few 1000 flops and memory accesses) and so I can't compute it on the fly. However I only need to compute it once and do not need to modify it later. I also can't assume that I have enough swap space on the system. That's why i decided to create a cache file and use mmap to map it to memory:

int createCacheFile(std::size_t filesize, std::string const& filename){
    //create empty file
    int fileDescriptor = open(filename.c_str(), O_RDWR | O_CREAT | O_TRUNC, (mode_t)0600);
    //stretch to desired size
    lseek(fileDescriptor, filesize-1, SEEK_SET);
    return fileDescriptor;
}

//...
std::size_t n = 100000;
std::size_t fileSize = n*n*sizeof(float);
int fileDescriptor = createCacheFile(filesize,"matrix.cache");
float* memory = (float*) mmap(0, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, fileDescriptor, 0);

//and now fill it...

I wanted to compare performance and tried a small n=10000 and compared malloc, mmap with MAP_ANONYMOUS and the above implementation. For this n the matrix fits completely into RAM. While malloc and MAP_ANONYMOUS give quite similar results, I get a roughly factor 10 penalty when computing my matrix when it is backed by a file. It seems that the program is regularly stopped by the kernel so that it can write the contents safely to the file. I tried to resolve this using msync and mprotect on the parts of the matrix that i have already computed to give the kernel a hint that it can write the sections without having to stop the programs but nothing helped.

Is there a way to fix this?

Answer 1

You could also use the madvise(2) syscall to inform the kernel about less useful pages (perhaps with MADV_SEQUENTIAL or MADV_DONTNEED ...). Perhaps the posix_fadvise(2) syscall might be helpful for the file segment. Eventually readahead(2) (in another thread, since it is blocking) might help also.

And the file might sit in a fast filesystem, perhaps a tmpfs one....

Perhaps swapping on a fast disk (SSD) might also be useful. swapon(2) syscall (and swapon command).