我要接受GPU训练吗？

Question

I'm training a neural model with keras and tensorflow as backend. The log file starts with the following message:

nohup: ignoring input
2019-02-12 17:44:29.414526: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-02-12 17:44:30.191565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:65:00.0
totalMemory: 7.93GiB freeMemory: 7.81GiB
2019-02-12 17:44:30.191601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2019-02-12 17:44:30.409790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-12 17:44:30.409828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 
2019-02-12 17:44:30.409834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N 
2019-02-12 17:44:30.410015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7535 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:65:00.0, compute capability: 6.1)

Does this mean that the training is performed on gpu ?

I would say yes, but when I execute nvtop , I see that all the gpu memory is used while 0% of the gpu calculation capacity is used (see yellow screenshot below):

Also, when I type htop in the command line, I see that one CPU is fully used (see black screenshot below)

How come the gpu memory is used and the cpu capacity calculation is used instead of the gpu capacity calculation ?

Answer 1

I think that you have compiled (or you installed already compiled package) tensorflow with CUDA support, but not with support all instructions available for your CPU (your CPU supports AVX2, AVX512F and FMA instructions that tensorflow can use).

This means, that tensorflow will work fine (with full GPU support), but you can't use your processor at full capacity.

Try compare time (GPU vs CPU) with this example: https://stackoverflow.com/a/54661896/10418812