Cannot fit model with all my images and masks, because of resources

Question

I have a U-Net NN and I have a dataset with 40k+ images (they are numpy arrays with shape 768x768x3) and 10k+ masks to this images(with shape 768x768x1). My computer and even Kaggle, Google Colab cannot fit it with their resources. But I need to fit the model with all of this images and masks. So, what do I need to do? I have think about make a loop with 400 images and 400 masks on each iterations and fit every time 400 images to my model, but I have read, that on each iteration model will 'reset'. So what I need to do to fit my model with all this images and masks?

Update : I have changed size of image and mask to (256,256,3) and (256,256,1), but even with this I can fit model with +/- 1000 images

Answer 1

This is a very common problem in Data Science. There is not one solution, it's basically a process.

Start out with the following steps and see if the problem still persists. Find out what is blocking, GPU or RAM.

• If GPU, you need to know that your model is loaded in GPU memory and the batches are put there iteratively. So, when GPU memory is the problem, reduce your model size, or the amount of data present in the batch.
• If RAM, you should look into DataLoaders if you use PyTorch or Sequence if you use Tensorflow/Keras. In general, the data is loaded like an iterator, meaning it's only really loaded upon request (with some sort of a waiting queue, you can also again specify how large that is allowed to be). This will make an epoch slower, as you cannot load all data upfront into memory for rapid access, but you still manage to iterate over your data and process it as such.