[英]How do you train GANs using multiple GPUs with Keras?
There are three main challenges: a) how do you save and load the optimizer state , b) how do you use multiple GPU with nested models, see below, and c), how you create a workflow to optimize GPU and CPU utilization? 有三个主要挑战:a)如何保存和加载优化器状态 ,b)如何使用多个GPU与嵌套模型,见下文,以及c),如何创建工作流以优化GPU和CPU利用率?
We have three components: 我们有三个组成部分:
Since the discriminators are included in the GAN and they also need to be used separately during training - how do you save and load GANs? 由于鉴别器包含在GAN中,它们也需要在培训期间单独使用 - 如何保存和加载GAN? Now, I save the generators and discriminators separately and recompile the GAN for each training episode, but I lose the optimizer state this way. 现在,我单独保存生成器和鉴别器,并为每个训练集重新编译GAN,但是我以这种方式失去了优化器状态。
This is what the API looks like: 这就是API的样子:
from keras.utils import multi_gpu_model
parallel_model = multi_gpu_model(model, gpus=8)
The challenge here is the same as with optimizers. 这里的挑战与优化者一样。 Since the discriminator is included in GANs, you can't apply the multi_gpu_model
to both the discriminator and the GAN. 由于鉴别器包含在GAN中,因此不能将multi_gpu_model
应用于鉴别器和GAN。 You can add a multi_gpu_model
to both the discriminator and generator before you create the GAN, but from my experience it does not scale well and leads to poor GPU utilization. 您可以在创建GAN之前向识别器和生成器添加multi_gpu_model
,但根据我的经验,它不能很好地扩展并导致GPU利用率低下。
The data can be preprocessed and queued using multiprocessing. 可以使用多处理对数据进行预处理和排队。 Since the multi_gpu_model
API does not support GANs, you need to frequently merge the weights and hop between CPUs and GPUs. 由于multi_gpu_model
API不支持GAN,因此您需要经常合并权重并在CPU和GPU之间跳转。 Thus, I haven't found a clean way to utilize GPUs and CPUs. 因此,我还没有找到一种利用GPU和CPU的简洁方法。
The multi_gpu_model can be used in each of the function for generator, discriminator and gan multi_gpu_model可用于发生器,鉴别器和gan的每个函数
def create_generator():
#network architecture
generator = Model(inputs=input, outputs=output)
generator = multi_gpu_model(generator, gpus=2)
generator.compile()
return generator
The same can be done for discriminator and gan. 鉴别器和甘也可以这样做。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.