Large vectors and memory conservation c++

Question

I'm working on making a tile-map editor in C++. Right now, while a map is being edited, its attributes are stored across three vectors:

vector<vector<vector<bool>>> CanvasCollisionObstruction; //[collision,obstruction][map x][map y]
vector<vector<vector<bool>>> CanvasZoneOverlays; //zone overlays for programmable map zones [zone type][map x][map y]
vector<vector<vector<canvasClip>>> CanvasClips; //identifies which sprite occupies this tile [layer number][map x][map y]

In the above vectors, the 2nd and 3rd dimensions ([map x] and [map y]) refer to the actual tile coordinates on the map. These are just plain-old-square-2d maps. The type of that last vector is the following struct:

struct canvasClip
{
    int tileset;
    int clip;
    //initialization to check if it's in-use
    canvasClip() : tileset(-1), clip(-1) {}
    bool isInitialized()
    {//a clip is only rendered if it is initialized
        return ((tileset >= 0) && (clip >= 0));
    }
    bool operator==(const canvasClip& a) const
    {//used in flood-fill to compare with target tile
        return ((tileset == a.tileset) && (clip == a.clip));
    }
    bool operator!=(const canvasClip& a) const
    {//used in flood-fill to compare with target tile
        return ((tileset != a.tileset) || (clip != a.clip));
    }
};

For this application, I expect to eventually want to generate maps with size upwards of 50000x50000 tiles, across indefinite (but probably never more than 10) layers. There are about 12 zones total, and that number is constant.

The map editor has some controls to change the size of the map (numeric inputs and a button). When I set the map size to a very large number, I call vector.resize() on each of those vectors, and I can watch my memory usage quickly travel upward in the task manager until my computer finally crashes.

Can anyone offer me some advice or tips for handling very large vectors? Am I going to have to do something like compress the vector so that a single index describes a span of similar tiles? Should I store the map in a file instead of memory, and then read-back only a few chunks of it at a time, as-needed?

How do good programmers handle this situation?

Answer 1

As it already mentioned in the comments, you are trying to allocate huge amount of memory only for data.

In this case, you have to choose different data structure to store it and operate with it.

Here is a couple of the simplest tricks that you may apply in the cost of complexity of the code that operates the data:

1. You have default values that seem to be meaningless. Why not to store in memory only the data that is set to true values?
2. You may store in memory only visible part of data (see Here's what's happening in Horizon: Zero Dawn every time you move a camera ).
3. You might have to pack your data structure and work on alignments (see Data structure alignment ).

Of course, there is a case when you have to limit requirements, but that's a part of a life.