如何将 stl 容器保存在磁盘中？

Question

I have a stl container, which save a lot of data.

for example, the definition is:

std::unordered_map<std::string, std::vector<int32_t> > m;  // there are 10 million data in this container

what i want is to save it in a file, then next time, i can read this file to recover it quickly.

i think the normal way, like:

write(&m, sizeof(m))

doesn't work, because it only save the pointer.

so, how can i save it in file and recover it by read file?

Answer 1

Choose a file format, eg a simple list, iterate over your container and write each element:

for (const auto &el : m) {
    file << el.first << '\n' << el.second.size() << '\n';
    for (const auto &vecEl : el.second) {
        file << vecEl << '\n';
    }
}

Then you can read the file with

std::string first;
std::size_t count;
while (std::getline(file, first) && file >> count) {
    std::vector<int32_t> v(count);
    for (auto &vecEl : v) {
        file >> vecEl;
    }
    std::cin.ignore(std::numeric_limits<std::streamsize>::max(), '\n');
    m[first] = v;
}

This is just a sketch. You probably have to add error checks.

Answer 2

Given you're not keen to use boost, the lowest-effort lowest-reusability option is something like:

// some support macros for easy error detection/handling...

#define FATAL(MSG) \
    do { \
         std::cerr << "FATAL ERROR at line " << __LINE__ << ":\n" \
             "    " << MSG << '\n'; \
         exit(EXIT_FAILURE); \
    } while (false)

#define ASSERT(X, MSG) \
     do { \
         if (X) break; \
         FATAL("ASSERT(" #X ") FAILED; " << MSG); \
     } while (false)

// output your container to disk...

if (std::ofstream out{filename}) {
    for (auto& [key, value] : m) {
        out << key.size() << ' ';
        out.write(key.data(), key.size();
        out << ' ' << value.size();
        for (auto& i : values)
            out << ' ' << i;
        out >> '\n';
    }
    ASSERT(out, "failed while writing to " << filename);
}
else
    FATAL("failed to create " << filename);

This just creates a file with lines like:

13 A 13 char str 5 1 2 3 4 5

The leading 13 means you know how many characters are in the string before parsing it, which saves us the trouble of coming up with some escaping and terminator convention, or the risk of not having one (eg assuming the string doesn't embed a newline is risky in general). The first '5' indicates the array length, only needed to speed loading, as it lets you reserve enough memory up front, which may be faster than using .push_back and having to resize during loading of an array.

Reading the file then becomes:

if (std::ifstream in{filename}) {
    size_t len;
    while (in >> len) {
        std::string key;
        key.resize(len);
        ASSERT(in.read(key.data(), len));
        ASSERT(in >> len);
        std::vector<int32_t> v;
        v.reserve(len);
        while (len--) {
            ASSERT(in >> num);
            v.push_back(num);
        }
    }
    ASSERT(out.eof(), "failed to parse string size");
    m[std::move(key)] = std::move(v);
}
else
    FATAL("failed to open " << filename);

I haven't tested this, so use/test/debug as needed....