Talos multi gpu feature

Question

Im trying to run a Talos hyperparameter search for my CNN. Having 6 GPU's to run an experiment faster, the Talos feature multi_gpu seems handy.

my_new_model = multi_gpu(my_new_model, gpus=6)
my_new_model.compile(...)

Calling the function before the model.compile is required.

But it seems, that still only one job is executed at the same time. I was hoping, that an experiment with 6 possible paths should be processed at the same time. Now its still working one after another.

Any ideas?

Answer 1

multi_gpu is for having a single task run on multiple GPUs on a single machine.

Depending on what you are doing in terms of dataset, architecture, parameters, etc. this can reduce experiment time significantly. Even though permutations are still performed "one by one", each permutation is performed in less time.