使用TensorFlow-GPU + Python多处理时出现错误？

Question

I have noticed a strange behavior when I use TensorFlow-GPU + Python multiprocessing.

I have implemented a DCGAN with some customizations and my own dataset. Since I am conditioning the DCGAN to certain features, I have training data and also test data for evaluation.

Due to the size of my datasets, I have written data loaders that run concurrently and preload into a queue using Python's multiprocessing .

The structure of the code roughly looks like this:

class ConcurrentLoader:
    def __init__(self, dataset):
        ...

class DCGAN
     ...

net = DCGAN()
training_data = ConcurrentLoader(path_to_training_data)
test_data = ConcurrentLoader(path_to_test_data)

This code runs fine on TensorFlow-CPU and on TensorFlow-GPU <= 1.3.0 using CUDA 8.0, but when I run the exact same code with TensorFlow-GPU 1.4.1 and CUDA 9 (latest releases of TF & CUDA as of Dec 2017) it crashes:

2017-12-20 01:15:39.524761: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2017-12-20 01:15:39.527795: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2017-12-20 01:15:39.529548: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2017-12-20 01:15:39.535341: E tensorflow/stream_executor/cuda/cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2017-12-20 01:15:39.535383: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2017-12-20 01:15:39.535397: F tensorflow/core/kernels/conv_ops.cc:667] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms) 
[1]    32299 abort (core dumped)  python dcgan.py --mode train --save_path ~/tf_run_dir/test --epochs 1

What really confuses me is that if I just remove test_data the error does not occur. Thus, for some strange reason, TensorFlow-GPU 1.4.1 & CUDA 9 work with just a single ConcurrentLoader , but crash when multiple loaders are initialized .

Even more interesting is that (after the exception) I have to manually shut down the python processes, because the GPU's VRAM, the system's RAM and even the python processes stay alive after the script crashes.

Furthermore, it has to have some weird connection to Python's multiprocessing module, because when I implement the same model in Keras (using TF backend!) the code also runs just fine, with 2 concurrent loaders. I guess Keras is somehow creating an layer of abstraction in between that keeps TF from crashing.

Where could I possibly have screwed up with the multiprocessing module that it causes crashes like this one?

These are the parts of the code that use multiprocessing inside the ConcurrentLoader :

def __init__(self, dataset):
    ...
    self._q = mp.Queue(64)
    self._file_cycler = cycle(img_files)
    self._worker = mp.Process(target=self._worker_func, daemon=True)
    self._worker.start()

def _worker_func(self):
    while True:
        ... # gets next filepaths from self._file_cycler
        buffer = list()
        for im_path in paths:
            ... # uses OpenCV to load each image & puts it into the buffer
        self._q.put(np.array(buffer).astype(np.float32))

...and this is it.

Where have I written "unstable" or "non-pythonic" multiprocessing code? I thought daemon=True should ensure that every process gets killed as soon as the main process dies? Unfortunately, this is not the case for this specific error.

Did I misuse the default multiprocessing.Process or multiprocessing.Queue here? I thought simply writing a class where I store batches of images inside a Queue and make it accessible through methods / instance variables should be just fine.

Answer 1

I am coming with the same error when trying to use tensorflow and multiprocessing

E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED

but in different environment tf1.4 + cuda 8.0 + cudnn 6.0. matrixMulCUBLAS in sample codes works fine. I wonder the correct solution too! And the reference failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED on a AWS p2.xlarge instance did not work for me.