使用帶有mpi4py的主機文件生成進程

Question

我正在嘗試使用MPI4py和OpenMPI在多個主機上生成一組輔助進程，但是spawn命令似乎忽略了我的主機文件。 我已經發布了完整的測試 ，但是下面是關鍵部分：

根據論壇討論 ，我的經理腳本使用hostfile選項調用spawn：

mpi_info = MPI.Info.Create()
mpi_info.Set("hostfile", "worker_hosts")

comm = MPI.COMM_SELF.Spawn(sys.executable,
                           args=['testworker.py'],
                           maxprocs=args.worker_count,
                           info=mpi_info).Merge()

在worker_hosts文件中，列出了Scyld Beowulf群集中的節點：

myhead1 slots=2
mycompute1 slots=2
mycompute2 slots=2
mycompute3 slots=2
mycompute4 slots=3

管理者和工作人員都調用MPI.Get_processor_name() ，但是他們都報告“ myhead1”。 如果我將相同的主機文件與mpirun一起使用，則可以使用：

> mpirun -hostfile worker_hosts -np 3 python -c "from mpi4py import MPI; print MPI.Get_processor_name()"
myhead1
myhead1
mycompute1

如果將主機文件的名稱更改為不存在的名稱（例如bogus_file ， bogus_file收到錯誤消息：

--------------------------------------------------------------------------
Open RTE was unable to open the hostfile:
    bogus_file
Check to make sure the path and filename are correct.
--------------------------------------------------------------------------
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file base/rmaps_base_support_fns.c at line 83
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file rmaps_rr.c at line 82
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file base/rmaps_base_map_job.c at line 88
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file base/plm_base_launch_support.c at line 105
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file plm_rsh_module.c at line 1173

因此，OpenMPI已經注意到hostfile選項，但似乎沒有使用它。 hostfile選項在OpenMPI文檔中列出。

Key                   Type      Description
---                   ----      -----------
host                  char *    Host on which the process should be spawned.
                                See the orte_host man page for an
                                explanation of how this will be used.
hostfile              char *    Hostfile containing the hosts on which
                                the processes are to be spawned. See
                                the orte_hostfile man page for an
                                explanation of how this will be used.

如何為生成請求指定主機文件？

Answer 1

我找到了OpenMPI文檔的更新版本，這給了我一個神奇的選擇：

Key                    Type     Description
---                    ----     -----------
host                   char *   Host on which the process should be
                                spawned.  See the orte_host man
                                page for an explanation of how this
                                will be used.
hostfile               char *   Hostfile containing the hosts on which
                                the processes are to be spawned. See
                                the orte_hostfile man page for
                                an explanation of how this will be
                                used.
add-host               char *   Add the specified host to the list of
                                hosts known to this job and use it for
                                the associated process. This will be
                                used similarly to the -host option.
add-hostfile           char *   Hostfile containing hosts to be added
                                to the list of hosts known to this job
                                and use it for the associated
                                process. This will be used similarly
                                to the -hostfile option.

如果我更改為使用add-hostfile ，則可以完美運行：

mpi_info.Set("add-hostfile", "worker_hosts")

如果您被困在使用舊版本的OpenMPI，請嘗試使用mpirun和相同的主機文件運行管理器腳本。 當我仍在使用hostfile選項時，這也起作用。

mpirun -hostfile worker_hosts -np1 python testmanager.py