Python: Parallel execution pysphere commands

Question

My current for loop does 1 by 1 removing snapshots from my 16 VMs

for vmName in vmList:
    snapshots = vmServer.get_vm_by_name(vmName).get_snapshots()
    for i in range(len(snapshots)-3):
        snapshotName = snapshots[i].get_name()
        print "Deleting snapshot " + snapshotName + " of " + vmName
        vmServer.get_vm_by_name(vmName).delete_named_snapshot(snapshotName)

I need to run it in parallel(so it wouldn't wait finish of previous job to start next one) I was trying to apply "multiprocessing", here's full code:

import argparse
from pysphere import VIServer # Tested with vCenter Server 5.5.0 and pysphere package 0.1.7
from CONFIG import * # Contains username and password for vCenter connection, list of VM names to take snapshot
from multiprocessing.pool import ThreadPool as Pool

def purgeSnapshotStage(vmList):
    # Connect to vCenter
    vmServer = VIServer()
    vmServer.connect("VM_ADDRESS", username, password)

    snapshots = vmServer.get_vm_by_name(vmName).get_snapshots()
    for i in range(len(snapshots) - 3):
        snapshotName = snapshots[i].get_name()
        print "Deleting snapshot " + snapshotName + "   of VM:   " + vmName
        vmServer.get_vm_by_name(vmName).delete_named_snapshot(snapshotName)

    vmServer.disconnect()

# Get the environment to delete snapshot from command line
parser = argparse.ArgumentParser(description="Take snapshot of VMs for stage or stage2")
parser.add_argument('env', choices=("stage", "stage2", "stage3"), help="Valid value stage or stage2 or stage3")
env = parser.parse_args().env
vmList = globals()[env + "VmList"]

pool_size = 5  # your "parallelness"
pool = Pool(pool_size)

for vmName in vmList:
    pool.apply_async(purgeSnapshotStage, (vmList,))

pool.close()
pool.join()

But there is a mistake, because it's trying to execute "remove" command only on last one. Didn't find good guide about multiprocessing, and can't find how to debug it. Need help to find mistake.

Answer 1

You have an error in here:

for vmName in vmList:
    pool.apply_async(purgeSnapshotStage, (vmList,))

It should be:

for vmName in vmList:
    pool.apply_async(purgeSnapshotStage, (vmName,))

And then in your function header you need this:

def purgeSnapshotStage(vmList):

Then, there might be other errors in your code.

Generally: I doubt that parallelizing this might give you any performance benefit. Your bottleneck will be the vmware server. It will not be faster when you start many delete jobs at the same time.