具有 LMDB 的内存数据库

Question

I have a project which uses BerkelyDB as a key value store for up to hundreds of millions of small records.

The way it's used is all the values are inserted into the database, and then they are iterated over using both sequential and random access, all from a single thread.

With BerkeleyDB, I can create in-memory databases that are "never intended to be preserved on disk". If the database is small enough to fit in the BerkeleyDB cache, it will never be written to disk. If it is bigger than the cache, then a temporary file will be created to hold the overflow. This option can speed things up significantly, as it prevents my application from writing gigabytes of dead data to disk when closing the database.

I have found that the BerkeleyDB write performance is too poor, even on an SSD, so I would like to switch to LMDB . However, based on the documentation, it doesn't seem like there is an option creating a non-persistent database.

What configuration/combination of options should I use to get the best performance out of LMDB if I don't care about persistence or concurrent access at all? ie to make it act like an "in-memory database" with temporary backing disk storage?

Answer 1

Just use MDB_NOSYNC and never call mdb_env_sync() yourself. You could also use MDB_WRITEMAP in addition. The OS will still eventually flush dirty pages to disk; you can play with /proc/sys/vm/dirty_ratio etc. to control that behavior.

Answer 2

From this post: https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/

vm.dirty_ratio is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk. When the system gets to this point all new I/O blocks until dirty pages have been written to disk.

If the dirty ratio is too small, then you will see frequent synchronous disk writes.