How to debug an error with python script with no traceback errors?

Question

def make_offsite_copies_primary():
    """multi-threaded function to update OFFSITE_PREFIX volumes to primary."""

    start_date = time.strftime("%m/%d/%y",
                           time.localtime(time.time() - DR_TAPE_DATE_RANGE))

    for pool in run_command([vmpool, "-listall", "-b"]).readlines():
        if pool.startswith(OFFSITE_PREFIX):
            pool = pool.split()[0]
            logentry("***\tupdating images for pool: %s" % (pool))
            command = [bpchangeprimary, "-pool", pool, "-sd", start_date]
            RunMTCommand.lck.acquire()
            # If we've reached maxthreads, then wait for one to finish
            if len(RunMTCommand.tlist) >= RunMTCommand.maxthreads:
                RunMTCommand.lck.release()
                RunMTCommand.evnt.wait()
            else:
                RunMTCommand.lck.release()
            # For some reason, we often get a "-15" RC, when updating the
            # MLC-Duplication pool.  Let's just ignore it for now.
            RunMTCommand.newthread(command, [0, 13, -15])

    for bpchange_thread in RunMTCommand.tlist:
        bpchange_thread.join()

This function issues a NetBackup command to promote a copy of a backup to be the primary copy.

When this script is run, it starts logging in the terminal as expected:

Mon Aug 24 10:04:08 2015 - 17684 - ***  updating images for pool: MLC-2week
Mon Aug 24 10:04:08 2015 - 17684 - ***  updating images for pool: MLC-1month
Mon Aug 24 10:04:08 2015 - 17684 - ***  updating images for pool: MLC-1year
Mon Aug 24 10:04:08 2015 - 17684 - ***  updating images for pool: MLC-Shadow-2week
Mon Aug 24 10:04:08 2015 - 17684 - ***  updating images for pool: MLC-2week-DR-Files
Mon Aug 24 10:04:08 2015 - 17684 - ***  updating images for pool: MLC-Offsite

etc.

However, the bpchangeprimary command does not seem to be called within the script because the backup copies do not change.

If you type 'bpchangeprimary -pool MLC-LTO4-Offsite -sd 08/22/2015' directly from the command line, this makes the intended changes to the backup copies. However, this line inside the function:

command = [bpchangeprimary, "-pool", pool, "-sd", start_date]

Should accomplish the same thing, but it's not, so I'm guessing something is wrong with this function. To make things worse, there are no traceback errors.

Any help, or tips on how to troubleshoot this would be greatly appreciated.

Answer 1

Actually command = [bpchangeprimary, "-pool", pool, "-sd", start_date] just prepares the command to be executed, it doesn't actually execute it.

Your logentry() call in its current position is therefore producing misleading log entries.

The command execution is only attempted at/after this call:

RunMTCommand.newthread(command, [0, 13, -15])

I'd start with moving your logentry() call (or adjusting its content and adding a new one) further down, below the if... then... else... logic meant to limit the number of threads - to first get meaningfull logging.

This should tell at least if the execution goes past the thread limiting logic:

• if it doesn't then you need to debug that logic
• if it does then focus on the thread starting in the multithread context and/or actual command execution