MemoryError: Unable to allocate 5.62 GiB for an array with shape (16384, 30720, 3) and data type float32 When training StyleGan2

Question

Training on tensorflow 1.15, python3.7. I am currently training stylegan2 on a custom dataset consisting of 30000 images, each 256 by 256. Since style gan creates different tf record files, each storing dataset with size 2^x. the 8th tf record (storign 256x256 images) is 5Gb just for context, the rest are super small (all less than a gb). My current setup is a P100, 16 gigs VRam, 32 gb ram and abundance of storage. I also 2vCpus (training on gcp).

I am running into this error as mentioned above. Initially my memory was 13 gigs, after seeing the exact error multiple times, i iteratively upped my memory to an eventual 32 gb.

Any and all "pointers" would be helpful (Notice the pun on pointers haha)

Answer 1

OKAY I SOLVED IT. there was an issue with the.pkl file that I was using for transfer learning. use a pickle file that contains a model whos discriminator starts with input layer of your image shape (eg 256x256).