MMAP storage Engine in Mongodb

Question

A storage engine acts as a interface which acts between the mongo db server and physical Disc which decides how much memory is required also supports Collection level locking. My question is what happened before version 3.0 ? Who allocated memory before the storage engine ? And how did the locking mechanism work before M MAP

Answer 1

There was only one storage engine before 3.0 - MMAP, which has been the storage engine for MongoDB since the beginning (now usually referred to as MMAPv0, with the version in 3.0 being MMAPv1, though the versioning is not really official like the DB itself).

You couldn't plug in new engines prior to 3.0 nor were there any alternatives built-in so you didn't see a lot of discussion about storage engines as a result. Any presentations (here's a good one if you are interested) prior to 3.0 that discuss storage are implicitly talking about the MMAP storage engine, it just didn't have that name yet.

MMAP has been improved to include collection level locking in 3.0, before that release (in 2.6) the locking granularity was database level and before that (prior to 2.2) it was a global lock.

Answer 2

We call it MMAPv1 - the original storage engine of MongoDB because it internally uses the mmap call under the covers in order to implement storage management. Let's look at what the MMAP system call looks like. On Linux, it talks about memory allocation, or mapping files or devices into memory. Causes the pages starting address and continuing for at most length bytes to be mapped from the object described by the file descriptor at an offset. So, what does that really practically mean?

Well, MongoDB practically needs a place to put documents. And it puts the documents inside files. And to do that it initially allocates, let's say a large file. Let's say it allocates a 100GB file on disk. So, we wind up with 100GB file on disk. The disk may or may not be physically contiguous on the actual disk, because there are some algorithms that occur beneath that layer that control the actual allocation of space on a disk. But from our point, it's a 100GB contiguous file. If MongoDB calls mmap system call, it can map this 100GB file into 100GB of virtual memory. To get this big virtual memory, we need to be on a x64 machine. And these are all page-sized . So pages on an OS or either 4k or 16k large. So, there is lot of them inside the 100GB virtual memory. And the operating system is going to decide what can fit in the memory. So, the actual physical memory of the box is let's say 32GB, then if we go to access one of the pages in this memory space, it may not be in memory at any given time. The operating system decides which of these pages are going to be in memory. We're showing the ones available in memory as green ones. So, when we go to read the document, if it hits a page that's in memory, then we get it. If it hits a page that's not in memory (the white ones), the OS has to bring it from the disk.

_source

MMAPv1 storage engine provides

1. Collection level concurrency (locking) . Each collection inside MongoDB is it's own file (can be seen in ~\\data\\db ). If multiple write s are fired on the same collection, one has to wait for another to finish. It's a multiple reader. Only one write can happen at a time to a particular collection.
2. Allows in place updates . So, if a document is sitting here in one of the available (green) page and we do an update to it, then we'll try to update it right in place. And if we can't update it, then what we'll do is we'll mark it as a whole, and then we'll move it somewhere else where there is some space. And finally we'll update it there. In order to make it possible that we update the document in place without having to move it, we uses

3. Power of 2 sizes when we allocate the initial storage for a document. So, if we try to create a 3bytes document, we'll get 4bytes. 8bytes, if we create 7bytes. 32bytes when creating 19bytes document. In this way, it's possible to grow the document a little bit. And that space that opens up, that we can re-use it more easily.

   Also, notice that since, OS decides what is in memory and what is on disk - we cannot do much about it. The OS is smart enough for memory management.