[英]Solving CUDA error: out of memory by code modification
I keep getting the following error while running this code on a server with GPU:在带有 GPU 的服务器上运行此代码时,我不断收到以下错误:
RuntimeError: CUDA out of memory. Tried to allocate 10.99 GiB (GPU 0; 10.76 GiB total capacity; 707.86 MiB already allocated; 2.61 GiB free; 726.00 MiB reserved in total by PyTorch)
I added a garbage collector.我添加了一个垃圾收集器。 I tried making the batch size really small (from 10000 to 10) and now the error has changed to:我尝试使批量大小非常小(从 10000 到 10),现在错误已更改为:
(main.py:2595652): Gdk-CRITICAL **: 11:16:04.013: gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed
2022-06-07 11:16:05.909522: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
File "main.py", line 194, in <module>
**psm = psm.cuda()**
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 637, in cuda
return self._apply(lambda t: t.cuda(device))
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 637, in <lambda>
return self._apply(lambda t: t.cuda(device))
**RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.**
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Here is a part of PMS.这是 PMS 的一部分。 I copied that as the error line shows psm = psm.cuda()
我复制了它,因为错误行显示psm = psm.cuda()
class PSM(nn.Module):
def __init__(self, n_classes, k, fr, num_feat_map=64, p=0.3, shar_channels=3):
super(PSM, self).__init__()
self.shar_channels = shar_channels
self.num_feat_map = num_feat_map
self.encoder = Encoder(k, fr, num_feat_map, p, shar_channels)
self.decoder = Decoder(n_classes, p)
def __call__(self, x):
return self.forward(x)
def forward(self, x):
encodes = []
outputs = []
for device in x:
encode = self.encoder(device)
outputs.append(self.decoder(encode.cuda()))
encodes.append(encode)
# Add shared channel
shared_encode = torch.mean(torch.stack(encodes), 2).permute(1,0,2).cuda()
outputs.append(self.decoder(shared_encode))
return torch.mean(torch.stack(outputs), 0)
This worked for me:这对我有用:
nvidia -smi
I found the GPU that was less busy.我发现 GPU 不太忙。 Then adding torch.cuda.set_device(1)
to my code.然后将torch.cuda.set_device(1)
添加到我的代码中。 I also used a reduced batch size or it will not work.我还使用了减小的批量大小,否则它将不起作用。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.