根据包含垃圾邮件术语的列表过滤元素

Question

So I've made a script that scrapes some sites and builds a list of results. Each result has the following structure:

result = {'id': id,
            'name': name,
            'url': url,
            'datetime': datetime,
        }

I want to filter results from the list of results based on spam terms being in the name. I've defined the following function, and it seems to filter certain results, but not all of them:

def filterSpamGigsList(theList):
    index = 0
    spamTerms = ['paid','hire','work','review','survey',
                 'home','rent','cash','pay','flex',
                 'facebook','sex','$$$','boss','secretary',
                 'loan','supplemental','income','sales',
                 'dollars','money']
    for i in theList:
        for y in spamTerms:
            if y in i['name'].lower():
                theList.pop(index)
                break        
            index += 1
    return theList

Any clue why this might not be filtering out all results that contain these spam terms? Maybe I need to call .split() on name after calling .lower() as some of the names are phrases?

Answer 1

I guess you've got a problem with in-place modifying theList as iterating over it as Jakub suggested.

The obious way would be to return a new list. I would split this in two functions for readability:

def is_spam(value):
    spam_terms = ['paid','hire','work','review','survey',
                 'home','rent','cash','pay','flex',
                 'facebook','sex','$$$','boss','secretary',
                 'loan','supplemental','income','sales',
                 'dollars','money']
    for term in spam_terms:
        if term in value.lower():
            return True
    return False

def filter_spam_gigs_list(results):
    return [i for i in results if not is_spam(i['name'])]