使用位集进行位图存储有什么好处？

Question

I'm currently evaluating whether I should utilize a single large bitset or many 64-bit unsigned longs (uint_64) to store a large amount of bitmap information. In this case, the bitmap represents the current status of a few GB of memory pages (dirty / not dirty), and has thousands of entries.

The work which I am performing requires that I be able to query and update the dirty pages, including performing OR operations between two dirty page bitmaps.

To be clear, I will be performing the following:

• Importing a bitmap from a file, and performing a bitwise OR operation with the existing bitmap
• Computing the hamming weight (counting the number of bits set to 1, which represents the number of dirty pages)
• Resetting / clearing a bit, to mark it as updated / clean
• Checking the current status of a bit, to determine if it is clean

It looks like it is easy to perform bitwise operations on a C++ bitset, and easily compute the hamming weight. However, I imagine there is no magic here -- the CPU can only perform bitwise operations on as many bytes as it can store in a register -- so the routine utilized by the bitset is likely the same I would implement myself. This is probably also true for the hamming weight.

In addition, importing the bitmap data from the file to the bitset looks ugly -- I need to perform bitshifts multiple times, as shown here . I imagine given the size of the bitsets I would be working with, this would have a negative performance impact. Of course, I imagine I could just use many small bitsets instead, but there may be no advantage to this (other then perhaps ease of implementation).

Any advice is appriciated, as always. Thanks!

Answer 1

Sounds like you have a very specific single-use application. Personally, I've never used a bitset, but from what I can tell its advantages are in being accessible as if it was an array of bools as well as being able to grow dynamically like a vector.

From what I can gather, you don't really have a need for either of those. If that's the case and if populating the bitset is a drama, I would tend towards doing it myself, given that it really is quite simple to allocate a whole bunch of integers and do bit operations on them.

Given that have very specific requirements, you will probably benefit from making your own optimizations. Having access to the raw bit data is kinda crucial for this (for example, using pre-calculated tables of hamming weights for a single byte, or even two bytes if you have memory to spare).

I don't generally advocate reinventing the wheel... But if you have special optimization requirements, it might be best to tailor your solution towards those. In this case, the functionality you are implementing is pretty simple.

Answer 2

I think if I were you I would probably just save myself the hassle of any DIY and use boost::dynamic_bitset . They've got all the bases covered in terms of functionality, including stream operator overloads which you could use for file IO (or just read your data in as unsigned int s and use their conversions, see their examples) and a count method for your Hamming weight. Boost is very highly regarded a least by Sutter & Alexandrescu, and they do everything in the header file--no linking, just #include the appropriate files. In addition, unlike the Standard Library bitset , you can wait until runtime to specify the size of the bitset.

Edit: Boost does seem to allow for the fast input reading that you need. dynamic_bitset supplies the following constructor:

template <typename BlockInputIterator>
dynamic_bitset(BlockInputIterator first, BlockInputIterator last,
               const Allocator& alloc = Allocator());

The underlying storage is a std::vector (or something almost identical to it) of Block s, eg uint64 s. So if you read in your bitmap as a std::vector of uint64 s, this constructor will write them directly into memory without any bitshifting.

Answer 3

Thousands bits does not sound as a lot. But maybe you have millions.

I suggest you write your code as-if you had the ideal implementation by abstracting it (to begin with use whatever implementation is easier to code, ignoring any performance and memory requirement problems) then try several alternative specific implementations to verify (by measuring them) which performs best.

One solution that you did not even consider is to use Judy arrays (specifically Judy1 arrays).