“NotFoundError：算法无效？”是什么意思？

Question

a time ago, I write a CNN code that detects cats and dogs. The data is taken from Kaggle and the CNN code is succesfully worked. And before that to make learning faster, I installed Nvidia Cuda. Now I found a new task from Kaggle ( https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia ). I have written CNN code again. But I get an error I have never seen.

Code:

import os
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Dense, Flatten
from keras import optimizers
from keras.preprocessing.image import ImageDataGenerator


def data():
    dataset = 'C:/Users/Lenovo/Desktop/chest_xray/'
    train = os.path.join(dataset, 'train')
    test = os.path.join(dataset, 'test')
    validation = os.path.join(dataset, 'val')
    train_data = ImageDataGenerator(rescale=1. / 255)
    test_data = ImageDataGenerator(rescale=1. / 255)

    train_generator = train_data.flow_from_directory(
        train,
        target_size=(1000, 1000),
        batch_size=16,
        class_mode='binary')

    validation_generator = test_data.flow_from_directory(
        validation,
        target_size=(1000, 1000),
        batch_size=16,
        class_mode='binary')

    test_generator = test_data.flow_from_directory(
        test,
        target_size=(1000, 1000),
        batch_size=16,
        class_mode='binary')

    for data_batch, labels_batch in train_generator:
        print('data batch shape:', data_batch.shape)
        print('label batch shape:', labels_batch.shape)
        break
    return validation_generator, test_generator, train_generator


def detector():
    model = Sequential()
    model.add(Conv2D(32, (4, 4), activation='relu', input_shape=(1000, 1000, 1)))
    model.add(MaxPooling2D((3, 3)))
    model.add(Conv2D(64, (4, 4), activation='relu'))
    model.add(MaxPooling2D((3, 3)))
    model.add(Conv2D(64, (4, 4), activation='relu'))
    model.add(MaxPooling2D((3, 3)))
    model.add(Conv2D(128, (4, 4), activation='relu'))
    model.add(MaxPooling2D((3, 3)))
    model.add(Conv2D(128, (4, 4), activation='relu'))
    model.add(MaxPooling2D((3, 3)))
    model.add(Flatten())
    model.add(Dense(256, activation='relu'))
    model.add(Dense(256, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy',
                  optimizer=optimizers.RMSprop(lr=1e-4))
    model.summary()
    return model


validation_, test_, train_ = data()
model_ = detector()

history = model_.fit_generator(
    train_,
    steps_per_epoch=100,
    epochs=30,
    validation_data=validation_,
    validation_steps=50)

Output:

C:\Users\Lenovo\anaconda3\python.exe "C:/Users/Lenovo/PycharmProjects/AI/1- Text.py"
2020-10-30 12:03:20.802426: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
Found 5216 images belonging to 2 classes.
Found 16 images belonging to 2 classes.
Found 624 images belonging to 2 classes.
data batch shape: (16, 1000, 1000, 3)
label batch shape: (16,)
2020-10-30 12:03:24.063892: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-10-30 12:03:25.009097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1050 Ti computeCapability: 6.1
coreClock: 1.62GHz coreCount: 6 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.43GiB/s
2020-10-30 12:03:25.009744: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-10-30 12:03:25.017137: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-10-30 12:03:25.022144: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-10-30 12:03:25.024962: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-10-30 12:03:25.030998: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-10-30 12:03:25.034229: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-10-30 12:03:25.052663: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-10-30 12:03:25.053092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-30 12:03:25.053825: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-30 12:03:25.063634: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1dafce31430 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-30 12:03:25.064420: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-30 12:03:25.065036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1050 Ti computeCapability: 6.1
coreClock: 1.62GHz coreCount: 6 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.43GiB/s
2020-10-30 12:03:25.065861: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-10-30 12:03:25.066244: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-10-30 12:03:25.066580: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-10-30 12:03:25.066872: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-10-30 12:03:25.067182: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-10-30 12:03:25.067619: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-10-30 12:03:25.068034: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-10-30 12:03:25.068448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-30 12:03:25.734018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-30 12:03:25.734464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-10-30 12:03:25.734664: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-10-30 12:03:25.735009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2984 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-10-30 12:03:25.738356: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1daa2f2ca40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-10-30 12:03:25.738781: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1050 Ti, Compute Capability 6.1
WARNING:tensorflow:From C:/Users/Lenovo/PycharmProjects/AI/1- Text.py:66: Model.fit_generator (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
Please use Model.fit, which supports generators.
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 997, 997, 32)      544       
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 332, 332, 32)      0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 329, 329, 64)      32832     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 109, 109, 64)      0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 106, 106, 64)      65600     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 35, 35, 64)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 32, 32, 128)       131200    
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 10, 10, 128)       0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 7, 7, 128)         262272    
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 2, 2, 128)         0         
_________________________________________________________________
flatten (Flatten)            (None, 512)               0         
_________________________________________________________________
dense (Dense)                (None, 256)               131328    
_________________________________________________________________
dense_1 (Dense)              (None, 256)               65792     
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 257       
=================================================================
Total params: 689,825
Trainable params: 689,825
Non-trainable params: 0
_________________________________________________________________
Epoch 1/30
2020-10-30 12:03:27.997615: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-10-30 12:03:28.951268: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-10-30 12:03:29.803616: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.91GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-10-30 12:03:29.804253: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2020-10-30 12:03:29.805214: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_ops.cc:1115 : Not found: No algorithm worked!
Traceback (most recent call last):
  File "C:/Users/Lenovo/PycharmProjects/AI/1- Text.py", line 66, in <module>
    history = model_.fit_generator(
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1815, in fit_generator
    return self.fit(
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1098, in fit
    tmp_logs = train_function(iterator)
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 840, in _call
    return self._stateless_fn(*args, **kwds)
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1843, in _filtered_call
    return self._call_flat(
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1923, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 545, in call
    outputs = execute.execute(
  File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.NotFoundError:  No algorithm worked!
     [[node sequential/conv2d/Conv2D (defined at /Users/Lenovo/PycharmProjects/AI/1- Text.py:66) ]] [Op:__inference_train_function_1397]

Function call stack:
train_function

2020-10-30 12:03:29.893708: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
     [[{{node PyFunc}}]]

Process finished with exit code 1

I search for that error but I didn't solve this. Is there any solution?

Answer 1

This might not be the answer for the current context. Anyway, the same error can occur when the input shapes mismatch. for an RGB image, there are 3 channels, 'R', 'G' and 'B' So, the input shape will be (height, width, 3) . If it is a grey scaled image the shape should be (height, width, 1)

model.add(Conv2D(
    32,
    (4, 4),
    activation='relu',
    input_shape=(1000, 1000, 1)
    )
)

Answer 2

I experienced this problem on Ubuntu with tensorflow on GPU.

The cause of the issue was problem with the cuda libraries.

In my case, it was resolved by installing nvidia-cuda-toolkit . In order to access these packages, you need the nvidia repo. in your sources.list :

sudo echo 'deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 /' >> /etc/apt/sources.list.d/nvidia-machine-learning.list
sudo apt update

Then install nvidia-cuda-toolkit :

# dependency required for nvidia-cuda-toolkit
sudo apt install nvidia-compute-utils-455
sudo apt install nvidia-cuda-toolkit