.. Microsoft C ++异常的未处理异常：内存位置处的std :: bad_alloc

Question

I have a large vector (mainvect) of struct info objects (about 8million element), and I want to remove duplicates, the struct consist of pid, and uid .

 struct info
 {
    int pid;
    string uid;

 }

I have another vector ( vect1 ) which contain information of each pid and its occurrence in mainvect (its help in search specific indices not all the main vect) size of vect1 is 420k elements

struct pidInfo
 {
    int pid;
    int numofoccurence;
}

I want to store unqiue elements in mainvect in vect2 .

 .
 .
// sort mainvect based on pid
sort(mainvect.begin(), mainvect.end(), sortByPId());

int start = 0;
int end = 0;
vector <string> temp;  // to store uids with a specific pid 

for (int i = 0; i < vect1.size(); i++)
{
   end = end + vect1[i].numofoccurence;

    for (int j = start; j < end; j++)
    {
        temp.push_back(mainvect[j].uid);
    }

    start = start + vect1[i].numofoccurence;

    // remove duplicate uid 
    sort(temp.begin(), temp.end());
    temp.erase(unique(temp.begin(), temp.end()), temp.end());

    // push remaining unique uids 
    for (int k = 0; k < temp.size(); k++)
    {
        info obb;
        obb.pid = vect1[i].pid;
        obb.uid = temp[k];
        vect2.push_back(obb);
    }

    // empty the temp vector to use in next i iteration
    temp.erase(temp.begin(), temp.end());

}
 .
 .

But when I run the code, it gave me exception as shown in the following figure

Answer 1

You are running out of memory. There are few things you can do:

1. You are building 32-bit program on Windows. This means you have only 2 GB of RAM available, regardles of how much physical memory have on your machine. You should build your program for 64-bit architecture to get access to all RAM you have. To do this you need to create of new configuration for your project with platform set to x64 .
2. You should be smarter about using your memory. The quickest thing you can do is to replace std::vector with std::dequeu for large vectors.

The problem with std::vector is that every time it grows it allocates a new memory chunk and copies all data. MSVC implementation you are using grows vector by factor of 1.5 each time. So if vector takes 1 GB of memory, next time it resizes it will try to allocate 1.5 GB, taking 2.5 GB of RAM in total while resizing is in progress.

Implementations of std::deque will usually allocate memory in smaller chunks, so it will have less problem with resizing.

Another thing you should pay attention to is std::string . MSVC implementation uses SSO (Small-String-Optimization) . Every instance of std::string` afair takes 32 bytes on x86. So every element in your 8 million elements vector might or might not be wasting this memory.

Depending on how much time you want to spend on your program, you might want to learn about memory-mapped files .

Answer 2

I think you actually have algorithm problem. On each iteration you are sorting and leaving only unique elements in temp vector. But with this approach each iteration will add more and more duplicates into vect2 . So you should sort and leave only unique elements in vect2 as well. Actually it would be probably better to utilize std::set instead of temp and vect2 .

Another suggestion would be to utilize a better storage for uid if it has some sort of fixes-length format, such as GUID.

Answer 3

As state above, you are running out of memory. If you really have so many elements, it may be a sensible idea to look into a small database, like sqlite.

But since the question is about C++ standard containers, you are approaching the problem a bit hamfisted. You are doing many unnecessary sorts and loops. Not only riddled with bugs, your algorithm is at least O(n^3)

Why not use one of the already sorted containers, for example std::map? You can deduplciate a list like so:

std::vector<info> input;

// copy into map
std::map<int, info> tmp;
for (info& i : mainvect) {
  tmp[i.pid] = i;
}

// copy back out
std::vector<info> output(tmp.size());
std::transform(tmp.begin(), tmp.end(), output.begin(), [] (const std::pair<int, info>& p) {
    return p.second;
});

Not only is the code cleaner, it runs at O(n + ln(n)). Or, skip the second step and use an std::map or std::set for the data in the first place.

Also if you handle a huge number of items, you don't want to use a std::vector. The key problem is that the memory for the vector needs to be one continuous piece of memory. You may want to use a deque or a list.