Ipython并行和多核流程

Question

I would like to use ipython parallel for job execution on a compute cluster. Setting things up is all relatively straightforward, but some jobs that I will be executing include subprocess calls to multithreaded applications (bioinformatics). Is there a simple way to tell ipython that multiple cores or entire engines are being consumed by a single python call? More generally, I'd like to track resources used and available so that I can run heterogeneous jobs (memory requirements, core requirements).

Answer 1

IPython does not do any resource allocation for individual tasks. All resource allocation is done at engine creation time. For instance, if your jobs are already fully multithreaded and multicore aware, then you may not want more than one IPython engine per physical machine. If your jobs are single-threaded, then one engine per CPU core is logical. If your tasks are IO-bound on multiple non-exclusive resources, then you may want more engines than cores.

If your work is highly heterogeneous (eg some tasks are confined to one thread, while others will happily use the whole machine), then you may want to arrange some more sophisticated scheduling. Two useful pieces of information for giving this a try are creating views of subsets of engines. For instance, a load-balanced and direct view with only one engine per machine in the cluster:

import socket
host_map = client[:].apply_async(socket.gethostname).get_dict()
r = { v:k for k,v in host_map.items() }
one_id_per_machine = list(r.values())
one_per_machine = client[one_id_per_machine]
lb_per_machine = client.load_balanced_view(one_id_per_machine)

And the other is the graph-based dependencies that you might use to create barriers, allowing one task to block others from being submitted to other engines.