Cannot write bitsets fast enough

Question

I need to be able to write down 12-bit bitsets on the order of speed of around 1 millisecond per 10,000 bitsets. Basically I'm provided with data in 12-bit packages (bitsets, in this case) and I need to be able to store them (I've chosen to write them to a file, open to suggestions if other methods exist) within an incredibly small timespan.

Right now I've set up an example of a bitset array of size 10,000 (to simulate what I would actually get) and write them all down into a file

int main()
{

    std::bitset<12> map[10000];

    std::ofstream os("myfile.txt", std::ofstream::binary);
    //From here
    for (int i = 0; i < 10000; ++i)
    {
        os << map[i];
    }
    //to here takes slightly under 7 ms -- too slow
}

As the comments say, it takes 7 ms. I'm open to any and all speed improvements, and am hopeful to get (optimally) 1ms for that loop.

Edit Info: This is for a Serial Peripheral Interface (SPI), and the data will be all available, as it is in the example, then dumped all at once, not as a stream of bitsets. For more technical specs, I'm using an Arduino Atmega328p, ADS7816, and an SD card reader

Answer 1

Two recommendations:

• Minimize trips to the OS. Write multiple bytes in one go.
• Pack the bits before writing. Your current solution writes the bits as characters, ie one byte for every bit. Write in binary mode, which would be 8 times more compact (and also faster).

#include <bitset>
#include <fstream>
#include <vector>

int main()
{
    std::bitset<12> map[10000];

    // Initialize with demo values
    for (int i = 0; i < 10000; ++i) {
        map[i] = i + 1;
    }

    // Pack bits into a binary buffer
    std::vector<uint8_t> buffer((10000 * 12 + 7) / 8);
    for (int i = 0, j = 0, rem = 0; i < 10000; ++i) {
        unsigned long b = map[i].to_ulong();
        buffer[j++] |= static_cast<uint8_t>(b >> (4 + rem));
        buffer[j] |= static_cast<uint8_t>(b << (4 - rem));
        rem += 12 % 8;
        if (rem >= 8) {
            rem -= 8;
            j++;
        }
    }

    // Write the buffer in 1 go
    std::ofstream os("myfile.bin", std::ofstream::binary);
    os.write(reinterpret_cast<const char*>(buffer.data()), buffer.size());
    os.close(); // don't forget to close() to flush the file
}

If you prefer to keep your text file format, at least enable buffering:

int main()
{
    std::bitset<12> map[10000];

    // Enable buffering
    std::vector<char> buf(256 * 1024);
    std::ofstream os("myfile.txt", std::ofstream::binary);
    os.rdbuf()->pubsetbuf(buf.data(), buf.size());

    for (int i = 0; i < 10000; ++i)
    {
        os << map[i];
    }
    os.close(); // to flush the buffer
}