How does Keras/Tensorflow use GPU and CPU?

Question

I want to know how the keras uses the resource of computer.

For example, suppose that we use the keras sequence class to train massive dataset, with 4 image input and 1 image output. The size of image is 640 x 480 each and the network has about 5M weights. If we set the training batch size as 4, where are the input images loaded( RAM or VRAM ) and where is the network is loaded?

I wonder this because, when I trained a network using 16 x (96 x 96 x 3) with batch size 32, network having 16M weights worked very well,

but I failed to train a network using 4 x (480 x 640 x 3) with batch size 3, network having 5M weights(Now my device is training with batch size 2). I cannot understand how Keras uses the resources so want to find the most efficient way to train the network with larger batch size.

My device has 128GB RAM and 32GB VRAM, with Nvidia Tesla V100.

Answer 1

This might be external problems with your local processes, apart from the memory usage of Keras.

1. Please upload the code you used for training.
2. Try monitoring the memory usage using the profiler tool .