Tensorflow GPU model.fit 崩溃 Z6FF9F4444AC481652F4412B5E16238

Question

I am trying to train a very basic model using tensorflow on the GPU (Spyder 4.1.5, Python 3.8.5, tensorflow 2.7.0).

It works fine on the CPU, but crashes if I set the device to GPU.

This is the code:

import tensorflow as tf
from tensorflow.keras import datasets, layers, models
import matplotlib.pyplot as plt

(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

train_images, test_images = train_images / 255.0, test_images / 255.0

class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))

model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))

model.summary()

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

with tf.device('gpu:0'):
    history = model.fit(train_images, train_labels, epochs=10, 
                    validation_data=(test_images, test_labels))

The kernel crashes at model.fit and the only output I get is:

" 2022 01:11:35.629875: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance‑critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022 01:11:36.017910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3987 MB memory: ‑> device: 0, name: NVIDIA GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5 "

I think tensorflow + cuda should be installed correctly, as I have another model trained on the GPU which works fine.

Is there any way I could get some more information about the crash? Could it be that it runs out of memory?

Pictures from the Spyder console during the GPU execution:

Before the crash it waits like this for some seconds:

And afterwards I get this:

Answer 1

I ran the script directly from the anaconda prompt. It was an installation problem with the cudnn. This was the raised error:

"I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8302 Could not load library cudnn_cnn_infer64_8.dll. Error code 126 Please make sure cudnn_cnn_infer64_8.dll is in your library path!"

Answer 2

In most of the case, RAM problems are given because you're not giving the right type of Data. Try convert your Dataset in:

• For X (Data) from list to numpy.array as following:

   import numpy as np array = np.asarray(array) # List could give you problems

• For Y (Labels) from list to categorical :

   from tensorflow.keras.utils import to_categorical y_tr = to_categorical(y_tr)

In your case before that you convert to_categorical, you need to convert your string_labels to int_labels so:

class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']

to

class_names = [0,1,2,3,4,5,6,7,8,9]

and then categorical that add 1 in the position of your classes and 0 in all the others so for example "horse" class will be:

[0,0,0,0,0,0,0,1,0,0]

for "automobile":

[0,1,0,0,0,...,0]