Weird behavior removing elements from a list in a loop in Python

Question

I understand that you can't remove an element from a list if you are currently iterating over that list. What I'm trying to do then is to copy elements from that list that I do NOT wish to remove to another list, and then replacing the original list with the new list. Here's my relevant code:

while len(tokenList) > 0:
    # loop through the tokenList list

    # reset the updated token list and the remove flag
    updatedTokenList = []
    removeFlag = False

    for token in tokenList:

        completionHash = aciServer.checkTaskForCompletion(token)

        # If the completion hash is not the empty hash, parse the information
        if completionHash != {}:
            # if we find that a task has completed, remove it from the list
            if completionHash['Status'] == 'FINISHED' and completionHash['Error'] == '':
                # The task completed successfully, remove the token from the list
                removeFlag = True

            elif completionHash['Status'] == 'RUNNING' and completionHash['Error'] == '':
                # The task must still be running
                print('Task ' + completionHash['Type'] + ' ' + token + ' has been running for ' + completionHash['Runtime'] + ' seconds.')

            elif completionHash['Status'] == 'queued':
                # The task is in the queue
                print('Task ' + completionHash['Type'] + ' ' + token + ' is queued in position ' + completionHash['QueuePosition'])

            elif completionHash['Status'] == 'not_found':
                # Did not find a task with this token, possible the task hasn't been added yet
                print(completionHash['Error'])

            # if the task is still running, no change to the token list will have occured

        else:
            # This is probably because the server got rid of the token after the task completed
            print('Completion hash is empty, something went wrong.')

            tokenListError.append(token)
            removeFlag = True

        if not removeFlag:
            print('appending token to updatedTokenList')
            updatedTokenList.append(token)


    print('length of tokenList after removal loop: ' + str(len(updatedTokenList)))

    # wait some time, proportional to the number of tasks left
    checkInterval = len(updatedTokenList) * checkIntervalMultiplier

    print('Waiting ' + str(checkInterval) + ' seconds before checking again...')
    print('Tokens remaining: ' + str(len(updatedTokenList)))

    # replace the original token list with the updated token list
    tokenList = updatedTokenList

    # wait a while based on how many tokens remain
    time.sleep(checkInterval)

So point of all this is to update the tokenList with new list. Every time through the loop, new tasks will have finished and they should NOT be added to the updatedTokenList. The remaining task tokens will and this replaces the original token list.

This does not work. On my first pass through, it does not add any tokens to the updatedTokenList even though no tasks have yet completed. I cannot figure out what I am doing wrong. Any suggestions?

Answer 1

This gets a lot easier if you move the logic into a function:

#This function should have a more descriptive name that follows your 
#project's API.
def should_keep(token):
    """returns True if the token should be kept"""
    #do other stuff here.  Possibly print stuff or whatever ...
    ...

Now, you can replace your list with a simple list comprehension:

tokenList = [ token for token in tokenList if should_keep(token) ]

Note that we haven't actually replaced the list. The old list could still presumably have references to it hanging around. If you want to replace the list in place though, it's no problem. We just use slice assignment:

tokenList[:] = [ token for token in tokenList if should_keep(token) ]

Answer 2

One problem is that you never set removeFlag to False after it first encounters a token that should be removed. As soon as it detects one that should be removed, it will also remove all tokens after that one from the list. You need to set it to False either in all your completionHash elifs (and make sure that the status values they test are the only possibilities) or just set it immediately inside your for token in tokenlist loop.

If in your tests the first job has finished by the time you first check for completion, that would match the described behavior.

Answer 3

I understand you wish to delete the items from the list without keeping them, so, what I think you can do is save the number that corresponds to the list items you want to delete. For example, lets say I have a list with numbers from 1 to 5, but I only want this list to get the odd numbers, so I want to delete the even ones. What I would do is set a loop with a counter, to check every item on the list for a condition (in this case I would check if myList[ItemNumber] % 2 == 0 ) and if it does, I would set ItemNumber as an item in another list. Then when all the items that are going to be deleted have their numbers in this new list, I would call another loop to run through this new list and delete the items from the other list which numbers are contained in the new one. Like so:

myList = [1, 2, 3, 4, 5]
count = 0
toBeDeleted = []
while count < len(myList):
    if myList[count] % 2 == 0:
        toBeDeleted.append(count)
    count += 1

cont2 = len(toBeDeleted)
while cont2 > 0:
    cont3 = toBeDeleted[cont2 - 1]
    myList.pop(cont3)
    cont2 -= 1

This worked just fine for this problem, so I believe and hope it will help you with yours.