如何阻止 CUDA 为每个训练 keras model 的子进程重新初始化？

Question

I am using CUDA/CUDNN to train multiple tensorflow keras models on my GPU (for an evolutionary algorithm attempting to optimize hyperparameters). Initially, the program would crash with an Out of Memory error after a couple generations. Eventually, I found that using a new sub-process for every model would clear the GPU memory automatically.

However, each process seems to reinitialize CUDA (loading dynamic libraries from the.dll files), which is incredibly time-consuming. Is there any method to avoid this?

Code is pasted below. The function "fitness_wrapper" is called for each indiv idual.

def fitness_wrapper(indiv):
    fit = multi.processing.Value('d', 0.0)
    if __name__ == '__main__':
        process = multiprocessing.Process(target=fitness, args=(indiv, fit))
        process.start()
        process.join()
    return (fit.value,)


def fitness(indiv, fit):
    model = tf.keras.Sequential.from_config(indiv['architecture'])
    optimizer_dict = indiv['optimizer']
    opt = tf.keras.optimizers.Adam(learning_rate=optimizer_dict['lr'], beta_1=optimizer_dict['b1'],
                                   beta_2=optimizer_dict['b2'],
                                   epsilon=optimizer_dict['epsilon'])
    model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
    model.fit(data_split[0], data_split[2], batch_size=32, epochs=5)
    fit = model.evaluate(data_split[1], data_split[3])[1]

Answer 1

Turns out the solution is to just use tensorflow.backend.clear_session() after each model (rather than the subprocesses). I had tried this before and it didn't work, but for some reason this time it fixed everything.

You should also delete the model and call reset_default_graph(), apparently.

def fitness(indiv, fit):
    model = tf.keras.Sequential.from_config(indiv['architecture'])
    optimizer_dict = indiv['optimizer']
    opt = tf.keras.optimizers.Adam(learning_rate=optimizer_dict['lr'], beta_1=optimizer_dict['b1'],
                                   beta_2=optimizer_dict['b2'],
                                   epsilon=optimizer_dict['epsilon'])
    model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
    model.fit(data_split[0], data_split[2], batch_size=32, epochs=5)
    fit = model.evaluate(data_split[1], data_split[3])[1]

    del model
    tf.keras.backend.clear_session()
    tf.compat.v1.reset_default_graph()

    return fit