使用多 GPU 训练时，Tensorflow 精度低

Question

Is it normal that I get lower accuracy when training on multi GPU? For example, when I train on single GPU with n batch size, I got 63% accuracy. However, when I train with 4 GPUs with n batch size for each GPU, I got only 58% accuracy. Both cases were trained with 100 epochs

I guess the averaging of gradient somehow makes it difficult for the optimizer. Anybody experienced the same thing ?

Answer 1

Since the model was trained after 100 epochs, you can infer the model and find the accuracy on CPU itself because it is not computationally heavy. But if you want to observe the accuracy while training, then it can be hard to find accuracy from each GPUs and averaging, which might not tell how good is the model that is being trained.