Keras - RTX 2080 ti training slower than both CPU-only and GTX 1070?

Question

I just got my 2080 ti today and hooked it right up to experiment with Keras on my models. But for some reason, when I train on a dense model the 2080 ti is 2 times slower than my CPU (an i7 4790k) and definitely slower than my old GTX 1070 (don't have exact numbers to compare it to).

To train one epoch on my CPU it takes 27 seconds while the 2080 ti is taking 67 seconds with nothing about the model or data changing. Same batch size of 128, etc. This is also significantly slower than my 1070 I just had in the machine last night. I checked the GPU usage while training and the memory usage goes up to max, and the GPU usage goes up to about 20%, while idle is 4%. I have CUDA 10, and the latest CuDNN on NVIDIA's site: v7.6.5. TensorFlow is 1.15

Does anyone have any clue what is going on here? If any more details are needed, just comment I can add them.

Answer 1

I figured it out, Thanks to the suggestion of a friend who got a 2060 recently, he noted that the default power mode is maximum power savings in the Nvidia Control Panel, or P8 power mode according to nvidia-smi (which is half clock speeds). After setting to prefer maximum performance in 3D settings, training times have significantly been reduced.

Answer 2

I also have problems with 1.15. Do you see an error like this by any chance?( https://github.com/tensorflow/models/issues/7640 ):

Internal: Invoking ptxas not supported on Windows Relying on driver to perform ptx compilation. This message will be only logged once.

Other people with this error have slow training as well.

Downgrading to Tensorflow 1.14 fixed it for me.