Writing to memory and to disk simultaneously

Question

I am developing an application which is required to keep a certain time t of data in memory (configurable) and another time t2 worth of data in persistent storage on disk. The reason behind this is so that frequently asked data will be stored in memory and retrieved faster, whilst older, less often user data will be stored and retrievable on disk.

The problem is: I can't simply write to memory and then copy the entire content of the memory buffer to disk after time t as if the application crashes, the most recent data stored in memory will be lost. So each time new data is received I need to store it simultaneously in memory and on disk.

My question is, is there an efficient way to mirror the buffer in memory to a portion of the disk? I am looking for a more efficient way than writing to memory and then to disk on each data update.

Answer 1

You can memory map the file, both unix and windows system support this (but with different API). After that, you can simply write to that memory location, and do a sync when needed.

Answer 2

Boost Memory-Mapped Files

The classes mapped_file_source, mapped_file_sink and mapped_file provide access to memory-mapped files on Windows and POSIX systems

LIVE DEMO

#include <boost/iostreams/device/mapped_file.hpp>
#include <boost/range/istream_range.hpp>
#include <boost/range/algorithm.hpp>
#include <boost/range/irange.hpp>
#include <iostream>
#include <iterator>
#include <fstream>
#include <string>

using namespace std;
using namespace boost;
using namespace boost::iostreams;

int main()
{
    string filename("test.mmap");
    { // Prepare file
        ofstream f(filename);
        copy(irange(0,255), ostreambuf_iterator<char>(f));
    }
    { // Work with memory mapped file
        mapped_file mm(filename);
        fill(mm, 0); // Works as range
        char *data=mm.data(); // Or as raw memory pointer
        copy("String in memory", data); // Copy to raw memory
    }
    { // print file contents
        ifstream f(filename);
        copy(istream_range<char>(f), ostream_iterator<char>(cout));
    }
}

Answer 3

The best method is to write to memory first then to the disk. If the memory is full, remove the least used item before writing to memory.

Another technique is to have a separate thread of execution that writes new items to the disk. The thread would wake up, check for new items and write those to the disk. The items would be marked as "not new". The thread then goes back to sleep. The sleep duration should be long enough so that more than one record is in memory, but fast enough so fewer records are will be lost during power off or crash.