memory mapped file access is very slow

Question

I am writing to a 930GB file (preallocated) on a Linux machine with 976 GB memory.

The application is written in C++ and I am memory mapping the file using Boost Interprocess. Before starting the code I set the stack size:

ulimit -s unlimited

The writing was very fast a week ago, but today it is running slow. I don't think the code has changed, but I may have accidentally changed something in my environment (it is an AWS instance).

The application ("write_data") doesn't seem to be using all the available memory. "top" shows:

Tasks: 559 total,   1 running, 558 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 98.5%id,  1.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  1007321952k total, 149232000k used, 858089952k free,   286496k buffers
Swap:        0k total,        0k used,        0k free, 142275392k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                  
  4904 root      20   0 2708m  37m  27m S  1.0  0.0   1:47.00 dockerd                                                                  
 56931 my_user   20   0  930g  29g  29g D  1.0  3.1  12:38.95 write_data                                                          
 57179 root      20   0     0    0    0 D  1.0  0.0   0:25.55 kworker/u257:1                                                           
 57512 my_user   20   0 15752 2664 1944 R  1.0  0.0   0:00.06 top

I thought the resident size (RES) should include the memory mapped data, so shouldn't it be > 930 GB (size of the file)?

Can someone suggest ways to diagnose the problem?

Answer 1

Memory mappings generally aren't eagerly populated. If some other program forced the file into the page cache, you'd see good performance from the start, otherwise you'd see poor performance as the file was paged in.

Given you have enough RAM to hold the whole file in memory, you may want to hint to the OS that it should prefetch the file, reducing the number of small reads triggered by page faults, substituting larger bulk reads. The posix_madvise API can be used to provide this hint, by passing POSIX_MADV_WILLNEED as the advice , indicating it should prefetch the whole file.