Fabric running commands on different hosts in parallel using threads

Question

I'm using fab to run commands on remote hosts. I can't use the built-in fab parallel execution mode because my code is invoked by some library which I don't have control over. It creates multiple threads and calls my driver(host) method with different host parameters.

Below is the code sample of how I'm trying to achieve it.

def setup_env_for_fab(host, user):
  env.host_string = host
  env.user = user

def run_command():
  run("python some_program.py")

def driver(host):
  setup_env_for_fab(host, "ubuntu")
  run_command()

So driver(host) function can be called by multiple threads with different host parameter. Will this run commands on different hosts parallely if driver(host) is called simultaneously by different threads? If this does not work as expected how can I achieve this?

Answer 1

Don't roll this by hand if you don't really need to. Fabric has the parallel execution built-in using multiprocessing , eg using the -P CLI flag:

$ fab -H web1,web2,web3 -P driver

See Fabric docs on parallel execution for all the options.

As for the threading support, see FAQ - Is Fabric thread-safe? . Current Fabric 1.10 is still not thread-safe:

Is Fabric thread-safe?

Currently, no, it's not – the present version of Fabric relies heavily on shared state in order to keep the codebase simple. However, there are definite plans to update its internals so that Fabric may be either threaded or otherwise parallelized so your tasks can run on multiple servers concurrently.

Update:

Fabric is now thread-safe: Upgrading from 1.x