在Python中对大图像数据集进行多处理

Question

I have a very large image dataset (>50G, single images in a folder) for training, to make loading of images more efficient, I firstly load parts of the images onto RAM and then send small batches to GPU for training.

I want to further speed up the data preparation process before feeding the images to the GPU and was thinking about multi-processing. But I'm not sure how should I do it, any ideas?

Answer 1

For speed I would advise to used HDF5 or LMDB:

I have successfully used ml-pyxis for creating deep learning datasets using LMDBs.

It allows to create binary blobs (LMDB) and they can be read quite fast. The link above comes with some simple examples on how to create and read the data. Including python generators/ iteratos

For multi-processing:

I personally work with Keras, and by using a python generator it is possible train with mutiple-processing for data using the fit_generator method.

fit_generator(self, generator, samples_per_epoch,
              nb_epoch, verbose=1, callbacks=[],
              validation_data=None, nb_val_samples=None,
              class_weight={}, max_q_size=10, nb_worker=1,
              pickle_safe=False)

Fits the model on data generated batch-by-batch by a Python generator. The generator is run in parallel to the model, for efficiency. For instance, this allows you to do real-time data augmentation on images on CPU in parallel to training your model on GPU. You can find the source code here , and the documentation here .

Answer 2

Don't know whether you prefer tensorflow/keras/torch/caffe whatever.

Multiprocessing is simply Using Multiple GPUs

Basically you are trying to leverage more hardware by delegating or spawning one child process for every GPU and let them do their magic. The example above is for Logistic Regression .

Of course you would be more keen on looking into Convnets - This LSU Material (Pgs 48-52[Slides 11-14]) builds some intuition

Keras is yet to officially provide support but you can "proceed at your own risk"

For multiprocessing, tensorflow is a better way to go about this (my opinion) In fact they have some good documentation on it too