在theano 0.9中对gpu使用3D卷积时的高CPU利用率

Question

My cpu utilization is 100% when using theano.tensor.nnet.conv3d I am adding conv3d_fft, convgrad3d_fft and convtransp3d_fft to my compiling mode in the theano function. The interesting part is my gpu utilization is also high. My data is all on gpu memory.

Any ideas about why it cpu utilization is so high?

Thanks!

Update: I tried the same convolution in Keras with Theano backend. It is shockingly faster than theano despite the fact that it also uses conv3d.

Answer 1

For invoking a task on the GPU, the CPU has to do some work (not negligible). If the tasks on the GPU are small, the tasks on the CPU become comparatively big. By the way, memory transfers are done by the DMA and do not cause CPU load except of invoking the transfer.

I had a CUDA program, where the CPU invoked a GPU task every ~100µs. Profiling the program I found out, that it was CPU bound, while nothing was done on CPU except invoking GPU tasks.