大型数据集的向量的替代方案？ C ++

Question

I am looking for a data structure that holds data in order that its inserted (like a vector) that needs to hold millions of unsigned longs. The key is that it needs to have a lookup thats better than O(logn), because it will get searched against a similar vector of the same size. Is there something that exists like this?

If I insert 10, 20, 30 and then iterate over the set, I need to guarantee the order of 10, 20, 30. My data is a string I converted into a unsigned long to reduce memory use, that is reverse decodable.

EDIT: Since people are asking, I am comparing two vectors against each other (both very large in size) to get the difference.

Small Example:

vector 1: 10 20 30 40 50 60

vector 2: 11 24 30 40 55 70 90

result:   30 40

Answer 1

I never used it myself and it might be out-of-date compared to recent C++ version features (last update is from 2011), but STXXL is meant to be a set of containers and algorithms built for very big amount of data. It might fit your need.

The core of STXXL is an implementation of the C++ standard template library STL for external memory (out-of-core) computations, ie, STXXL implements containers and algorithms that can process huge volumes of data that only fit on disks. While the closeness to the STL supports ease of use and compatibility with existing applications, another design priority is high performance.

The key features of STXXL are:

• Transparent support of parallel disks. The library provides implementations of basic parallel disk algorithms. STXXL is the only external memory algorithm library supporting parallel disks.
• The library is able to handle problems of very large size (tested to up to dozens of terabytes).
• Improved utilization of computer resources. STXXL implementations of external memory algorithms and data structures benefit from overlapping of I/O and computation.
• Small constant factors in I/O volume. A unique library feature called "pipelining" can save more than half the number of I/Os, by streaming data between algorithmic components, instead of temporarily storing them on disk. A development branch supports asynchronous execution of the algorithmic components, enabling high-level task parallelism.
• Shorter development times due to well known STL-compatible interfaces for external memory algorithms and data structures.
• STL algorithms can be directly applied to STXXL containers; moreover, the I/O complexity of the algorithms remains optimal in most of the cases.

For internal computation, parallel algorithms from the MCSTL or the libstdc++ parallel mode are optionally utilized, making the algorithms inherently benefit from multi-core parallelism.

Answer 2

A hash map is one way you will have faster lookup than a sorted vector. You must have c++11 support to use it.
http://www.cplusplus.com/reference/unordered_map/unordered_map/
To preserve the order of the data the only way would be to maintain a vector beside it that stored the int's as well
Before you jump to using it you should consider how you are going to use this data structure (access pattern). Also consider what the data you will be getting is likely to be.
Here is boost's version of the same thing http://www.boost.org/doc/libs/1_53_0/doc/html/unordered.html

Answer 3

I think what you should use is unordered_map combined with maybe a doubly-linked list for the order.

So every time you add a new item to your database you add it first to the front (or the end) of the linked list, and then add it to the hashmap where the key is the value (the unsigned int) and the "value" (from the key/value pair) is the pointer to the the object in the linked list. So now if you want a fast lookup you look in the hashmap, and if you want to iterate by order you use the linked list. Of course when you want to remove an object you have to remove them from both, but complexity wise it's the same (O(1) amortized for everything).

This will of course increase your memory by 2 or 3 compared to just using a hashmap.