Use mmap for random read from file but the read throuput is not as expected

Question

I write kv data into file with value size of 100KB and key size of 10Bytes. Only value data is written into file and indices of keys are stored in ram. I write 10M keys and I get a 1TB value file. I use nvme ssd and 100G memory. Then I use mmap to read data from the file. I get a read throughput of 1.8GB/s which is calculated by my metrics code. But my disk read throughput is 3.2GB/s, which is observed by iostat . I think this is caused by prefetch . So I use madvise to close prefetch as follows.

  const void* p_val_buffer_ = mmap(NULL, val_buffer_size_, PROT_READ,
                       MAP_PRIVATE | MAP_POPULATE | MAP_NONBLOCK, fd, 0);
  if (nullptr == p_val_buffer_) return -2;

  int ret =
      madvise(const_cast<void*>(p_val_buffer_), val_buffer_size_, MADV_RANDOM);
  if (0 != ret) {
    return ret;
  }

I get the same read throuput but the throughput is only 600MB/s. This is far below the limit performance of NMVE SSD. Anyone help me?

Answer 1

I found the problem. It is because I keep using 10 threads to read data when I use MADV_RANDOM . That is not enough to make a deep io depth. I use 100 threads and I can get read throughput of 3.2GB/s