在训练卷积神经网络（DenseNet）时，是否有任何选项或参数可以改变以减少训练时间？

Question

I am using the following python code 'ImageAI' to use DenseNet for my research.

ImageAI github: https://github.com/OlafenwaMoses/ImageAI

ImageAI example: https://towardsdatascience.com/train-image-recognition-ai-with-5-lines-of-code-8ed0bdd8d9ba

I am currently doing research related to symbol recognition (2D building drawing symbols) using the CNN approach (DenseNet). One example of a symbol of a VAV box is: https://ibb.co/cyhwRvf

I am trying to classify 39 classes(number of objects which is the number of symbols in this case) and have 2,000 images of each class for training data(2,000 x 39 = 78,000). And I have 1,000 images of each for the test data(1,000 x 39 = 39,000). The total size of the dataset is 1.82 GB (I consider this as a relatively small size but please correct me if I am wrong).

But the problem is that the training time is taking so much time.

I have tried using the GPU (Nvidia Geforce RTX 2080 Ti) and it is taking 3 days to train when I set the epoch(number of experiments) to be 200.

I would like to know whether if there is a way to reduce the time for training. Is there any parameters that I can change or any other options?

Or is this considered normal consumption time considering the size of the dataset and the GPU that I am using?

The five lines of code for training are the following:

from imageai.Prediction.Custom import ModelTraining

model_trainer = ModelTraining()

model_trainer.setModelTypeAsDenseNet()

model_trainer.setDataDirectory("mechsymbol")
model_trainer.trainModel(num_objects=39, num_experiments=200, enhance_data=True, batch_size=32, show_network_summary=True)

Answer 1

Im fairly new to ImageAI/Tensorflow but still learning.

In terms of getting a faster learning using parameters, I think the enhance_data will get you a bit faster learning by setting it to False. This parameter should be set to True if you dont have such a large set of images to train on (less than 1000). This parameter will more samples from existing ones.

enhance_data = False

And also worth mention is that you don not have to do all 200 generation training if you dont get any better results after for example 50 epoch due to overfitting you model.

Have heard about "mixed precision traning" using FP16 instead of FP32?

I have not tried it myself since it would be a marginal difference on a Nvidia GTX 1080 Ti since it does not have tensor cores. But you can read more about it here https://medium.com/tensorflow/automatic-mixed-precision-in-tensorflow-for-faster-ai-training-on-nvidia-gpus-6033234b2540

For reference:

When I do my ResNet training of a image data set with 62 classes (26GB) which has between 1000 and 2000 images for training and 200 for validation it takes about 10 minutes for each epoch.

That would be about 33 hours for a 200 generation training.

This is my imageAi python lines of code

model_trainer = ModelTraining()
model_trainer.setModelTypeAsResNet()
model_trainer.setDataDirectory('.')
model_trainer.trainModel(num_objects=classes_in_train, num_experiments=100, 
enhance_data=False, batch_size=32)

Found 81086 images belonging to 62 classes.
Found 13503 images belonging to 62 classes.

Epoch 38/100
231/2533 [=>............................] - ETA: 10:00 - loss: 2.9065e-04 - acc: 0.9999

Hope it might give you helpful info.