allocating 3 dimensional vectors of size 10000 10000 3 in c++

Question

I'm making a picture editing program, and I'm stuck in allocating memory. I have no idea what is going on.

Ok.. So when I do this:

std::vector<unsigned char> h;
for (int a = 0; a < 10000 * 10000 * 3; a++) {
    h.push_back(0);
}

this is fine(sorry I had to), but when I do this:

std::vector<std::vector<std::vector<unsigned char>>> h;
for (uint32_t a = 0; a < 10000; a++) {
  h.push_back({});
  for (uint32_t b = 0; b < 10000; b++) {
    h.at(a).push_back({});
    for (uint32_t c = 0; c < 3; c++) {
      h.at(a).at(b).push_back(0xff);
    }
  }
}

my memory usage explodes, and I get error: Microsoft C++ exception: std::bad_alloc at memory location 0x009CF51C

I'm working with .bmp.

Currently, code is in testing mode so it's basically a giant mess... I'm 15, so don't expect much of me.

I was searching for solutions, but all I found was like how to handle large integers and so on...

If you can give me maybe another solution, but I want my code to be as beginner friendly as it can get.

Answer 1

This is due to overhead of vector<char> . Each such object with 3 elements takes not 3 bytes, but probably 4 (due to reallocation policy), plus 3 pointers which probably take 3*8=24 bytes. Overall your structure takes 9.3 times the memory it could have.

If you replace the inner vector with an array, it will start working, since array does not have this overhead:

std::vector<std::vector<std::array<unsigned char, 3>>> h;
for (uint32_t a = 0; a < 10000; a++) {
  h.emplace_back();
  for (uint32_t b = 0; b < 10000; b++) {
    h.at(a).emplace_back();
    for (auto &c : h.at(a).at(b)) {
      c = 0xff;
    }
  }
}

Another alternative is to put the smaller dimension first.

Answer 2

My guess would be that the memory is being heavily fragmented by the constant vector reallocation, resulting in madness. For data this large, I would suggest simply storing a 1-dimensional pre-allocated vector:

std::vector h(10000 * 10000 * 3);

And then come up with an array accessing scheme that takes the X/Y arguments and turns them into an index in your 1d array, eg.:

int get_index(int x, int y, int width) {
    return ((y * width) + x) * 3;
}

If the image size is always fixed, you can also use std::array (see multi-dimensional arrays ), since the size is defined at compile-time and it won't suffer the same memory issues as the dynamically allocated vectors.

Answer 3

I don't know if this will help your problem, but you could try allocating the memory for the vec of vecs of vecs all at the beginning, with the constructor.

std::vector<std::vector<std::vector<unsigned char>>> h(10000, std::vector<std::vector<unsigned char>>(10000, std::vector<unsigned char>(3,0xff)));

BTW, you're getting a good start writing C++ at 15! I didn't start studying computer science till I was in my 20s. It really is a very marketable career path, and there are a lot of intellectually stimulating, challenging things to learn. Best of luck!