如何在HPC群集上通过python使用所有分配的节点

Question

I have a HPC cluster with SLURM installed. I can properly allocate nodes and cores for myself. I would like to be able to use all the allocated cores regardless of the node they are in. As i seen in this thread Using the multiprocessing module for cluster computing this cannot be achieved with multiprocessing .

My script look like this (oversimplified version):

def func(input_data):
    #lots of computing
    return data

parallel_pool = multiprocessing.Pool(processes=300)
returned_data_list = []
for i in parallel_pool.imap_unordered(func, lots_of_input_data)
    returned_data_list.append(i)
# Do additional computing with the returned_data
....

This script works perfectly fine, however as i mentioned multiprocessing is not a good tool for me, as even if SLURM allocated 3 nodes for me, multiprocessing can only use one. As far as i understand this is a limitation of multiprocessing.

I could use the srun protocol of SLURM, but that ust executes the same script N times, and i need additional computing with the output of the parallel processes. I could of course store the outputs somewhere, and ream em back in, but there must be some more elegant solution.

In the mentioned thread there are suggestions like jug , but as i was reading through it i havet found a solution for myself.

Maybe py4mpi can be a solution for me? The tutorials for that seems very messy, and i havent found a specific solution for my problem in there neither. (run a function in parallel with mpi, and then continue the script).

I tried subprocess calls, but the seem to work the same way as multiprocess calls, so they only run on one node. I havent found any confirmation of this, so this is only from my trial-and-error guess.

How can i overcome this problem? Currently i could use more than 300 cores, but one node only have 32, so if i could find a solution then i would be able to run my project nearly 10 times as fast.

Thanks

Answer 1

很多麻烦的后scoop是解决我的问题库。