[英]Why GPU memory-usage is quite different when using multi-gpu in tensorflow?
I'm using Tensorflow 1.4.0, two gpus training. 我正在使用Tensorflow 1.4.0,两个GPU训练。
Why two gpu memory usage is quite different? 为什么两个GPU的内存使用情况大不相同? Here's the gpu situation: 这是GPU的情况:
+-------------------------------+----------------------+----------------------+
| 4 Tesla K80 On | 00000000:00:1B.0 Off | 0 |
| N/A 50C P0 70W / 149W | 8538MiB / 11439MiB | 100% E. Process |
+-------------------------------+----------------------+----------------------+
| 5 Tesla K80 On | 00000000:00:1C.0 Off | 0 |
| N/A 42C P0 79W / 149W | 4442MiB / 11439MiB | 48% E. Process |
+-------------------------------+----------------------+----------------------+
Gpu memory used in GPU4 is twice than GPU5. GPU4中使用的Gpu内存是GPU5的两倍。 I think gpu memory used in both gpus should be about the same . 我认为两个GPU中使用的GPU内存应该大致相同。 Why is this situation? 为什么会这样呢? Is anyone help me ? 有人帮我吗? Thanks a lot! 非常感谢!
Here's the code and two functions to compute average gradients: 这是代码和两个函数来计算平均梯度:
tower_grads = []
lossList = []
accuracyList = []
for gpu in range(NUM_GPUS):
with tf.device(assign_to_device('/gpu:{}'.format(gpu), ps_device='/cpu:0')):
print '============ GPU {} ============'.format(gpu)
imageBatch, labelBatch, epochNow = read_and_decode_TFRecordDataset(
args.tfrecords, BATCH_SIZE, EPOCH_NUM)
identityPretrainModel = identity_pretrain_inference.IdenityPretrainNetwork(IS_TRAINING,
BN_TRAINING, CLASS_NUM, DROPOUT_TRAINING)
logits = identityPretrainModel.inference(
imageBatch)
losses = identityPretrainModel.cal_loss(logits, labelBatch)
accuracy = identityPretrainModel.cal_accuracy(logits, labelBatch)
optimizer = tf.train.AdamOptimizer(learning_rate=LEARNING_RATE)
grads_and_vars = optimizer.compute_gradients(losses)
lossList.append(losses)
accuracyList.append(accuracy)
tower_grads.append(grads_and_vars)
grads_and_vars = average_gradients(tower_grads)
train = optimizer.apply_gradients(grads_and_vars)
global_step = tf.train.get_or_create_global_step()
incr_global_step = tf.assign(global_step, global_step + 1)
losses = sum(lossList) / NUM_GPUS
accuracy = sum(accuracyList) / NUM_GPUS
def assign_to_device(device, ps_device='/cpu:0'):
def _assign(op):
node_def = op if isinstance(op, tf.NodeDef) else op.node_def
if node_def.op in PS_OPS:
return ps_device
else:
return device
return _assign
def average_gradients(tower_grads):
average_grads = []
for grad_and_vars in zip(*tower_grads):
# Note that each grad_and_vars looks like the following:
# ((grad0_gpu0, var0_gpu0), ... , (grad0_gpuN, var0_gpuN))
grads = []
for g, _ in grad_and_vars:
# Add 0 dimension to the gradients to represent the tower.
expanded_g = tf.expand_dims(g, 0)
# Append on a 'tower' dimension which we will average over below.
grads.append(expanded_g)
# Average over the 'tower' dimension.
grad = tf.concat(grads, 0)
grad = tf.reduce_mean(grad, 0)
# Keep in mind that the Variables are redundant because they are shared
# across towers. So .. we will just return the first tower's pointer to
# the Variable.
v = grad_and_vars[0][1]
grad_and_var = (grad, v)
average_grads.append(grad_and_var)
return average_grads
Multi gpu code come from : multigpu_cnn.py . 多个gpu代码来自: multigpu_cnn.py 。 The reason is that line 124, with tf.device('/cpu:0'):
is missed! 原因是with tf.device('/cpu:0'):
第124行! In this case, all ops are placed on GPU0. 在这种情况下,所有操作都放置在GPU0上。 So memory cost on gpu0 is much more than the others. 因此,gpu0上的内存成本比其他内存成本高得多。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.