在使用 tensorflow 库训练深度学习模型时出现错误：ResourceExhaustedError OOM on gpu(128 gb RAM) 请帮助我

Question

C:\\Users\\CVL-Acoustics\\Documents\\bangla-sentence-correction-master>python train.py Sit back and relax, it will take some time to train the model... Vocabulary size 250000 WARNING:tensorflow:From C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\rnn.py:417: calling reverse_sequence (from tensorflow.python.ops.array_ops) with seq_dim is deprecated and will be removed in a future version. Instructions for updating: seq_dim is deprecated, use seq_axis instead WARNING:tensorflow:From C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\util\\deprecation.py:432: calling reverse_sequence (from tensorflow.python.ops.array_ops) with batch_dim is deprecated and will be removed in a future version. Instructions for updating: batch_dim is deprecated, use batch_axis instead WARNING:tensorflow:From train.py:228: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

epoch 1 training Traceback (most recent call last): File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py", line 1322, in _do_call return fn(*args) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py", line 1307, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py", line 1409, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[6656,250000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Reshape, Variable_1/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: rnn/while/cond/Add/_87 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_421_rnn/while/cond/Add", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_clooprnn/while/cond/ArgMax/dimension/_1)]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train.py", line 321, in _, l = sess.run([train_op, loss], fd) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py", line 900, in run run_metadata_ptr) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py", line 1316, in _do_run run_metadata) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py", line 1335, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[6656,250000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU :0"](Reshape, Variable_1/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: rnn/while/cond/Add/_87 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_421_rnn/while/cond/Add", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_clooprnn/while/cond/ArgMax/dimension/_1)]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'MatMul', defined at: File "train.py", line 218, in decoder_logits_flat = tf.add(tf.matmul(decoder_outputs_flat, W), b) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py", line 2014, in matmul a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\gen_math_ops.py", line 4278, in mat_mul name=name) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py", line 3414, in create_op op_def=op_def) File "C:\\Users\\CVL-Acoustics\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py", line 1740, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[6656,250000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Reshape, Variable_1/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: rnn/while/cond/Add/_87 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_421_rnn/while/cond/Add", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_clooprnn/while/cond/ArgMax/dimension/_1)]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Answer 1

There are several reasons why this would happen.

• Try reducing the parameters of the network.
• Try decreasing your batch size.
• Check if another kernel is currently active that is allocating the memory.