最近 ImageAI 更新后自定义 Object 检测训练期间几乎没有 GPU 使用

Question

I have trained a custom object detection model using 750 images using ImageAI on Google Colab Pro about a month ago using TensorFlowGPU 1.13 and have roughly 30min/epoch training time. Now, when I train using the same dataset but with TensorFlowGPU 2.4.3 (ImageAI doesnt support old TF anymore), I am getting very little GPU usage (0.1GB) and 6 hour per epoch training times. I have tried training the same model on my local machine and I am getting very slow training times as well.

I am using the following imports (based on ImageAI documentation):

.pip install tensorflow-gpu==2.4.0 keras==2.4.3 numpy==1.19.3 pillow==7.0.0 scipy==1.4.1 h5py==2.10.0 matplotlib==3.3.2 opencv-python keras-resnet==0.2.0 !pip install imageai --upgrade

I am pulling my training data from Google Drive.

Is there anything I could be missing that could speed up my object detection training times on either Google Colab or my local machine? The slow training times is slowing my research down.

Answer 1

If you want full GPU usage, from my experience, you must revert back to previous versions of ImageAI and it's compatible packages. Here is a list of compatible packages that I have installed that work as of now (January 2021) on my local machine and Google Colab:

• TF-GPU==1.13.1
• Keras==2.2.4
• Imageai==2.1.0

This fixed any issue caused by the most recent patch of ImageAI. I now am back to full GPU usage. Until the issue is patched, I suggest using the old version.