CNTK：cuDNN故障7：CUDNN_STATUS_MAPPING_ERROR

Question

我正在嘗試從224 * 244 * 3圖像向下到和56 * 56 * 3張量創建一個簡單的卷積，我想與其他圖像進行比較。

為此，我創建了一個復合閱讀器

scale = ImageDeserializer.scale(width=224,
                                height=224,
                                channels=3,
                                scale_mode="pad",
                                pad_value=114,
                                interpolations='linear')
scale2 = ImageDeserializer.scale(width=56,
                                 height=56,
                                 channels=3,
                                 scale_mode="pad",
                                 pad_value=114,
                                 interpolations='linear')
image_source = ImageDeserializer(os.path.join(path, "images_map.txt"))
image_source.ignore_labels()
image_source.map_features('features', [scale])

mask_source = ImageDeserializer(os.path.join(path, "images_mask_map.txt"))
mask_source.ignore_labels()
mask_source.map_features('mask', [scale2])

return MinibatchSource([image_source, mask_source])

並與該讀者一起創建輸入地圖

input_map = {
    input_var: reader_train["features"],
    input_var_mask: reader_train["mask"]
}

CNN看起來像這樣

conv1 = cntk.layers.Convolution((5, 5), filterdims[0], pad=True, activation=cntk.ops.relu)(input_var)
maxpool1 = cntk.layers.MaxPooling((2, 2), (2, 2))(conv1)
conv2 = cntk.layers.Convolution((4, 4), filterdims[1], pad=True, activation=cntk.ops.relu)(maxpool1)
maxpool2 = cntk.layers.MaxPooling((2, 2), (2, 2))(conv2)
conv3 = cntk.layers.Convolution((4, 4), 3, pad=True, activation=cntk.ops.relu)(maxpool2)
return conv3 # shape is (3, 56, 56) conv3 = z in the error equation

有輸入

input_var = cntk.ops.input_variable((3, 224, 224), np.float32)
input_var_mask = cntk.ops.input_variable((3, 56, 56), np.float32)

和誤差函數

f2 = cntk.ops.element_times(cntk.ops.constant(0.00390625), input_var_mask, name="f2")
err = cntk.ops.reshape(cntk.ops.minus(z, f2), (56 * 56 * 3))
sq_err = cntk.ops.element_times(err, err)
mse = cntk.ops.reduce_mean(sq_err)
rmse_loss = cntk.ops.sqrt(mse)
rmse_eval = cntk.ops.sqrt(mse)

當我訓練時，一切正常

 data = reader_train.next_minibatch(min(minibatch_size, epoch_size - sample_count), input_map=input_map)  # fetch minibatch.
 trainer.train_minibatch(data)  # Error as in title 

我在哪里

cuDNN failure 7: CUDNN_STATUS_MAPPING_ERROR ; GPU=0 ; hostname=STEPHENPC
train_minibatch_overload_for_minibatchdata

return _cntk_py.Trainer_train_minibatch_overload_for_minibatchdata(self, *args)
RuntimeError: cuDNN failure 7: CUDNN_STATUS_MAPPING_ERROR ; GPU=0 ;      hostname=STEPHENPC ; expr=err

cudaStreamDestroy failed (PrefetchGPUDataTransferer dtor): an illegal  memory access was encountered (cuda error 77)

可以幫助我並告訴我錯誤原因嗎？

提前致謝

Answer 1

根據遇到此問題的其他項目（例如，請參閱此處 ），這可能表明cudnn庫中存在錯誤。 CNTK不使用紋理內存，因此最好將此問題報告給NVidia。