如何修复 NVIDIA Tegra TX2 上的 tensorflow 错误“GPU 同步失败”

Question

I am trying to run a prediction of a model build in keras on my NVIDIA Tegra TX2 using Tensorflow and Python (2.7) and I am quite randomly running in tensorflow giving me the following exception:

Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4504 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2) 2019-10-04 16:17:50.786531: E tensorflow/stream_executor/cuda/cuda_driver.cc:1032] could not synchronize on CUDA context: CUDA_ERROR_UNKNOWN: unknown error:: *** Begin stack trace *** stream_executor::gpu::GpuDriver::SynchronizeContext(stream_executor::gpu::GpuContext*) stream_executor::StreamExecutor::SynchronizeAllActivity() tensorflow::GPUUtil::SyncAll(tensorflow::Device*) *** End stack trace ***

...

tensorflow.python.framework.errors_impl.InternalError: GPU sync failed

Sometimes after a few reboots / waiting time the problem is solved and I can run the prediction again but 8 out of 10 times this error appears.

I've already tried the following:

• Change the query amount and memory usage as follows:

 config = tf.ConfigProto() config.gpu_options.allow_growth = True config.per_process_gpu_memory_fraction = 0.7 session = tf.Session(config=config, ...)

• Re-Install Tensorflow build for TX2 and Jetpack v3.3

I would be really happy for any further suggestions.

Answer 1

Since this problem is intermittent, this may be happening when tensorflow is not getting the required memory. It is a known issue and you have already tried the basic trouble shooting steps. Since you are still getting the issue, try below steps as well:

reinstall libhdf5-dev, python-h5py

sudo apt-get install libhdf5-dev
sudo apt-get install python-h5py

and then set gpu allow growth as per " https://github.com/keras-team/keras/issues/4161#issuecomment-366031228 "

import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
set_session(sess)

Answer 2

Adding the following line solved the issue. Note my TensorFlow version is 2.1>

import tensorflow as tf
config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.compat.v1.InteractiveSession(config=config)