[英]How to import a local python module when using the sbatch command in SLURM
I was using the cluster manager slurm and I was running a submission script with sbatch (with a python interpeter). 我正在使用集群管理器slurm ,我正在使用sbatch运行提交脚本(使用python interpeter)。 The sbatch submission imported one of my modules called
main_nn.py
. sbatch提交导入了我的一个名为
main_nn.py
模块。 The module is located in the same place as my submission directory, however, python fails to find it even though the file exists. 该模块与我的提交目录位于同一位置,但是,即使该文件存在,python也无法找到它。 I am having a hard time figuring it out why this is happening.
我很难搞清楚为什么会发生这种情况。 My python file looks as follow:
我的python文件如下所示:
#!/usr/bin/env python
#SBATCH --job-name=Python
print('hi')
import main_nn
however the output of my slurm dump file is: 但是我的slurm转储文件的输出是:
hi
Traceback (most recent call last):
File "/home/slurm/slurmd/job3223398/slurm_script", line6, in <module>
import main_nn
ImportError: No module named main_nn
I tried checking if the module main_nn
was in the current directory and it was there indeed. 我试着检查模块
main_nn
是否在当前目录中,确实存在。 Thus, the first thing that seemed suspicious to me was that the error in the slurm file said the location of my script was at "/home/slurm/slurmd/job3223398/slurm_script"
rather than at path_to_project
. 因此,对我来说似乎可疑的第一件事是slurm文件中的错误说我的脚本的位置是在
"/home/slurm/slurmd/job3223398/slurm_script"
而不是在path_to_project
。 Thus I went ahead an added the line 因此我继续增加了这条线
os.system('pwd')
to see where my script was executing from and to my surprise it was executing at path_to_project
rather than at "/home/slurm/slurmd/job3223398/slurm_script"
which must mean that sbatch is doing something funky to executed a script at one location but make it think its at another. 看看我的脚本执行的位置,令我惊讶的是它在
path_to_project
执行而不是在"/home/slurm/slurmd/job3223398/slurm_script"
,这必然意味着sbatch正在做一些时髦的事情,以便在一个位置执行脚本但是它认为它在另一个。 If this is the case how am I suppose to do an import in python where the module is in the same location as in my submission script? 如果是这种情况我怎么想在python中进行导入,其中模块与我的提交脚本位于同一位置? Am I forced to put it in a package and trick python to think its in a package/library?
我是否被迫将它放在一个包中并欺骗python以在包/库中进行思考?
As Slurm copies the submission script to a specific location on the compute node to run it, your Python script will not find the modules that are in the submission directory. 当Slurm将提交脚本复制到计算节点上的特定位置以运行它时,您的Python脚本将找不到提交目录中的模块。
But Slurm correctly sets the current working directory so you can explicitly add it to the python path with something like: 但是Slurm正确设置了当前的工作目录,因此您可以使用以下内容将其显式添加到python路径:
sys.path.append(os.getcwd())
near the beginning of your script. 靠近脚本的开头。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.