在 keras 中使用 multi_gpu_model 时出现值错误

Question

I am using google cloud VM with 4 Tesla K80 GPU's.

I am running a keras model using multi_gpu_model with gpus=4(since i have 4 gpu's). But, i am getting the following error

ValueError: To call multi_gpu_model with gpus=4 , we expect the following devices to be available: ['/cpu:0', '/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3']. However this machine only has: ['/cpu:0', '/xla_cpu:0', '/xla_gpu:0', '/gpu:0']. Try reducing gpus .

I can see that there are only two gpu's here namely '/xla_gpu:0', '/gpu:0' . so, i tried with gpus = 2 and again got the following error

ValueError: To call multi_gpu_model with gpus=2 , we expect the following devices to be available: ['/cpu:0', '/gpu:0', '/gpu:1']. However this machine only has: ['/cpu:0', '/xla_cpu:0', '/xla_gpu:0', '/gpu:0']. Try reducing gpus .

can anyone help me out with the error. Thanks!

Answer 1

It looks like Keras only sees one of the GPUs.

Make sure that all 4 GPUs are accessible, you can use device_lib with TensorFlow.

from tensorflow.python.client import device_lib

def get_available_gpus():
    local_device_protos = device_lib.list_local_devices()
    return [x.name for x in local_device_protos if x.device_type == 'GPU']

You might need to manually install or update GPU drivers on your instance. Consult here .

Answer 2

TensorFlow is only seeing one GPU (the gpu and xla_gpu devices are two backends over the same physical device). Are you setting CUDA_VISIBLE_DEVICES? Does nvidia-smi show all GPUs?

Answer 3

I had the same issue and I think I figured out a way around it. In my case, I am working on an HPC and I intalled keras on my /.local, whereas Tensorflow and CUDA are installed by the IT staff, anyway I encountered the same error above. I am using Tensorflow==1.15.0 and Keras==2.3.1

I noticed that the message error:

ValueError: To call multi_gpu_model with gpus=2, we expect the following devices to be available: ['/cpu:0', '/gpu:0', '/gpu:1']. However this machine only has: ['/cpu:0', '/xla_cpu:0', '/xla_gpu:0', '/xla_gpu:1']. Try reducing gpus.

is located in the following file of keras, line 184:

/home/.local/lib/python3.7/site-packages/keras/utils/multi_gpu_utils.py

I solved this, by replacing the line 175 with the following:

target_devices = ['/cpu:0'] + ['/gpu:%d' % i for i in target_gpu_ids] (before)
target_devices = ['/cpu:0'] + ['/xla_gpu:%d' % i for i in target_gpu_ids] (after)

Moreover, I modified the following keras file:

/home/.local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py

so I replaced the line 510 with:

return [x for x in _LOCAL_DEVICES if 'device:gpu' in x.lower()] (before)
return [x for x in _LOCAL_DEVICES if 'device:XLA_GPU' in x] (after)

Long story short only to say that apparently this is a bug of Keras and not of some environmental setup. After such modification my network was able to run with the xla_gpus, I hope this is somehow helpful.

Answer 4

You can check all device list using following code:

from tensorflow.python.client import device_lib
device_lib.list_local_devices()

Answer 5

This can be caused by using tensorflow instead of tensorflow-gpu .

One way to fix this is the following:

$ pip uninstall tensorflow
$ pip install tensorflow-gpu

More information can be found here: https://stackoverflow.com/a/42652258/6543020

Answer 6

I had the same issue. Tensorflow-gpu 1.14 installed, CUDA 10.0 , and 4 XLA_GPUs were displayed with device_lib.list_local_devices() .

I have another conda environement and there is just Tensorflow 1.14 installed and no tensorflow-gpu, and i don't know why, but i can run my multi_gpu model on all gpus with that environment.