如何创建内存映射的C ++对象列表

Question

It's been a while since I have done any C++, and I'm a little rusty on the best way to implement the following:

We have a database which stores large amounts of "objects". I am trying to think of a way to load an entire list of these objects into memory, but because of the size and number of these objects, it would be impractical to actually have them all in memory. Rather, I would like to have a "memory mapped file-like" system where the objects are loaded on demand when they are accessed. In other words, let the OS or something similar manage which objects should be in memory, similar to how the OS decides which segments of a file should be paged into memory. Can anyone give .me a hint on how this could be done?

Answer 1

If you're rusty on C++, you might take a simple approach.

You mention "Objects"; I take this as "user data" not as actual serialized c++ classes.

Anyhow, A memory mapped file is just a file. You're going to be reading from the file, the OS isn't going to solve your problems for you.

My advice, is to keep it simple. Implement your "objects" as a normal file I/O. Then once you have that working, you can bump up the perfomance by using a memory-mapped file instead.

As for design patterns, I would design an CObject class that gets created, by the CDataBase class. The CDataBase would know where every Object in the file (Database) is, then would create CObjects as is needed (reading them from the file).

Good luck.

Answer 2

Just a warning - if you have a large number of objects stored in SQL tables that you want to arbitrarily load into memory, it will likely be slow in multiple ways: many hits to the database (try use a minimum number of queries), too many constructor calls (use memory pools), etc...

...but you need to take it a step at a time - see if you can read an record from SQL into an instantiated object first. The best speed optimization will be in how you organize your data to minimize hits to the DB and to minimize constructor calls.

Note that a memory-mapped solution would be in lieu of a SQL table - it will be faster, but less flexible than SQL and you will have the trouble of double-maintenance: data in SQL must be kept in-sync with your memory-mapped file.

Answer 3

查看Boost内存映射文件 。

Answer 4

So depending on the size of your objects you could store them in a Hash map which keeps a LRU list and starts evicting objects to a file while keeping the key and a file offset in the map. That way when you do need to pull them back from disk it is one seek and a read to get the object back. If you then want to go to preallocating files of a fixed size you could mmap them and your offset would become just another pointer.

This is a simplification of how the Riak database's initial storage worked, they have some design docs on it over on their website [1][2]. This only works really well if your objects are larger than the keys such that all the keys fit in memory easily but the objects don't.

The Cassandra database uses a similar technique with its "key cache" [3].

You could also look to something like Berkly DB for your local store

[1] : http://wiki.basho.com/Concepts.html#Data-Storage
[2] : http://downloads.basho.com/papers/bitcask-intro.pdf
[3] : http://www.datastax.com/dev/blog/maximizing-cache-benefit-with-cassandra