如何在多个 GPU 的 Pytorch 示例中利用 DistributedDataParallel 的世界大小参数？

Question

I am running this Pytorch example on a g2.2xlarge AWS machine. So, when I run time python imageNet.py ImageNet2 , it runs well with the following timing:

real    3m16.253s
user    1m50.376s
sys 1m0.872s

However, when I add the world-size parameter, it gets stuck and does not execute anything. The command is as follows: time python imageNet.py --world-size 2 ImageNet2

So, how do I leverage the DistributedDataParallel functionality with the world-size parameter in this script. The world-size parameter is nothing but number of distributed processes .

Do I spin up another similar instance for this purpose? If yes, then how do the script recognize the instance? Do I need to add some parameters like the instance's IP or something?

Answer 1

World size argument is the number of nodes in your distributed training, so if you set the world size to 2 you need to run the same command with a different rank on the other node. If you just want to increase the number of GPUs on a single node you need to change ngpus_per_node instead. Take a look at the multiple node example in this Readme .