在 cpu 上并行运行多个神经网络

Question

I am trying to run multiple instances of same neural network training in pytorch but with different hyperparameters. For example different learning rate, optimizer etc. I tried to create a multiprocessing.Pool but I am getting an error:

Runtime error: Unable to handle autograd's threading in combination with fork-based multiprocessing. See https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork

Here is the sample code to give an idea of what I am doing:

def run_one_instance(idx=0):
    lr = choose_random([0.001, 0.01, 0.1, 1])
    optimizer = choose_random(["SGD", "Adam"])
    num_hidden_layers = choose_random([2,3,4,5])
    model = create_model(num_hidden_layers, lr, optimizer)
    model.train()
    return model.evaluate()

def run_multiple_instances(num_instances=40):
    pool_obj = multiprocessing.Pool()
    result = pool_obj.map(run_one_instance, range(0, num_instances))
    return result

So is there something I am doing wrong? Is there any other way to run multiple neural networks in parallel (on cpu or gpu)?

Answer 1

As the link in the error message explains , use Python's multiprocessing's spawn context instead of the fork context. Like so:

import multiprocessing as mp


ctx = mp.get_context('spawn')


def run_one_instance(idx=0):
    lr = choose_random([0.001, 0.01, 0.1, 1])
    optimizer = choose_random(["SGD", "Adam"])
    num_hidden_layers = choose_random([2,3,4,5])
    model = create_model(num_hidden_layers, lr, optimizer)
    model.train()
    return model.evaluate()

def run_multiple_instances(num_instances=40):
    pool_obj = ctx.Pool()
    result = pool_obj.map(run_one_instance, range(0, num_instances))
    return result

The ctx object has the same API as the multiprocessing module. See the Python multiprocessing docs for more info about multiprocessing contexts (eg spawn and fork ).