python multiprocessing google compute engine

Question

I have a python script which I run on my old quad-core laptop through multiprocessing. The simulation cannot be parallelised, I just run different instances of it on the different cores.

I was considering renting some more powerful cpus on google compute engine. Will I be able to use the same python multiprocessing script just with more cores available?

the script just calls a pool and then apply_async many times

Answer 1

Yes it should be possible. If you need access to multiple cores on the same instance, as an example the App Engine flexible runtime , which uses Compute Engine as the underlying VM, allows you to configure the number of cores available

Answer 2

Multiprocessing works the same way on Google's VMs as on your local box (roughly...). In order to use all cores available on a box, do not pass processes argument to multiprocesing.Pool , like:

pool = multiprocessing.Pool(processes=None)

This will make Python create a Pool with multiprocessing.cpu_count() number of processes, cpu_count usually being a number of cores.