Python index error in loop when index is in for loop

Question

I have a very strange problem in a function I created which searches for words in categories and deletes this category if the word is not inside it. And for some very mysterious reasons I always get the error:

list index out of range

I understand what this error means but I am unable to understand why. My code is the following:

def check_cat(input, list_of_words, categories):
"""if a word is not in the possible set of words of a class, cannot be in this class"""
possible_cat = list_of_words
categories_copy = categories
for j in range(len(list_of_words)):
    for i in input:
        if i not in list_of_words[j][:,1]:
            possible_cat.pop(j)
            categories_copy = np.delete(categories_copy,j)
        else:
            pass

where categories = array(['culture', 'politics', 'sports'], dtype='|S8') and

list_of_words =    
[array([['0.14285714285714285', 'ball'],
            ['0.2857142857142857', 'cart'],
            ['0.14285714285714285', 'drama'],
            ['0.14285714285714285', 'opera'],
            ['0.2857142857142857', 'theater']], dtype='|S32'),
     array([['0.25', 'decision'],
            ['0.5', 'drama'],
            ['0.25', 'strategy']], dtype='|S32'),
     array([['0.2857142857142857', 'ball'],
            ['0.14285714285714285', 'cart'],
            ['0.2857142857142857', 'goal'],
            ['0.14285714285714285', 'player'],
            ['0.14285714285714285', 'strategy']], dtype='|S32')]

The thing that I really don't understand is that when I execute the code 'outside' the function/without function, it works. But through a function, I get the error:

    File "<ipython-input-110-b499e8f5d937>", line 7, in check_cat
    if i not in list_of_words[j][:,1]:
IndexError: list index out of range

It seems to me that the index j is in the range of list_of_words because I loop inside it... Any help would be extremely appreciated.

Answer 1

I think the source of your error lies in your variable assignment. When you were assigning your two variables inside of the function, you aren't actually creating a copy but rather creating a link to the original Python object. So when you do pop it is actually reducing the length of the original one while the len was read first so the loop executes more times than there are numbers of items.

This is a great article to read more in-depth about what I explained and is one of the things you must keep in mind to avoid future pitfalls.

As for your question, I stopped getting the error with a small change where I copied the inputs rather than create a reference to it with .copy() .

possible_cat = list_of_words.copy()
categories_copy = categories.copy()

Hopefully, this clears it up and is what you're looking for.