gpucompute* 在 slurm 集群中关闭*

Question

我的 gpucompute 节点处于关闭状态，无法在 GPU 节点上发送作业。 在遵循网络上的所有解决方案后，我无法返回我的“关闭 GPU”节点。 在出现此问题之前，在通过运行“NVIDIA-Linux-x86_64-410.79.run --no”解决该错误后，我遇到了 Nvidia 驱动程序配置错误，无法通过“nvidia-smi”检测到 GPU -drm' 我遇到了这个错误，这是因为节点处于关闭状态。 非常感谢任何帮助！

command: sbatch md1.s
sbatch: error: Batch job submission failed: Requested node configuration is not available

command:  sinfo
PARTITION   AVAIL  TIMELIMIT  NODES  STATE NODELIST
gpucompute*    up   infinite      1  down* fwb-lab-tesla1

command:  sinfo -R
REASON               USER      TIMESTAMP           NODELIST
Not responding       slurm     2020-09-25T13:13:19 fwb-lab-tesla1

 command: sinfo -Nl
Fri Sep 25 16:35:25 2020
NODELIST        NODES   PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON              
fwb-lab-tesla1      1 gpucompute*       down*   32   32:1:1  64000        0      1   (null)Not responding 


command: vim /etc/slurm/slurm.conf
# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ControlMachine=FWB-Lab-Tesla
#ControlAddr=137.72.38.102
#
MailProg=/bin/mail
MpiDefault=none
#MpiParams=ports=#-#
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
#SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
#SlurmUser=slurm
SlurmdUser=root
StateSaveLocation=/var/spool/slurm/StateSave
SwitchType=switch/none
TaskPlugin=task/cgroup
#
#
# TIMERS
#KillWait=30

command: ls /etc/init.d
functions  livesys  livesys-late  netconsole  network  README

command: nvidia-smi
Fri Sep 25 16:35:01 2020    

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN V             Off  | 00000000:02:00.0 Off |                  N/A |
| 24%   32C    P8    N/A /  N/A |      0MiB / 12036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN V             Off  | 00000000:03:00.0 Off |                  N/A |
| 23%   35C    P8    N/A /  N/A |      0MiB / 12036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN V             Off  | 00000000:83:00.0 Off |                  N/A |
| 30%   44C    P8    N/A /  N/A |      0MiB / 12036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  TITAN V             Off  | 00000000:84:00.0 Off |                  N/A |
| 31%   42C    P8    N/A /  N/A |      0MiB / 12036MiB |      6%      Default |
---------------------------------------------------------------------------+
                                                                               
----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |

Answer 1

这个问题你提到可能防止slurmd上守护gpucompute启动。 您应该能够通过运行systemctl status slurmd或 Linux 发行版的等效命令来确认这一点。

slurmd日志可能包含类似于

slurmd[1234]: fatal: can't stat gres.conf file /dev/nvidia0: No such file or directory

尝试重新启动它

systemctl start slurmd

一旦确定nvidia-smi响应正确。

Answer 2

我的问题通过以下说明解决了。 请记住，无论何时重新启动系统，都需要在重新启动后输入命令。 感谢 Joan Bryan 解决了这个问题！

slurmd -Dcvvv
reboot
ps -ef | grep slurm
kill xxxx (this is Process id number in the output of previous ps ef command)
nvidia-smi
systemctl start slurmctld
systemctl start slurmd
scontrol update nodename=fwb-lab-tesla1 state=idle

now you can run the jobs on the GPU nodes!
Cheers