Python: Removing elements from a list inside an iterator?

Question

I am trying to remove elements from a list in Python. Most answers seem to suggest that using a list iterator is best, but I don't think it's possible (or at least elegant) for my problem.

I want to iterate over the test_data list and delete any items that meet the following two conditions: (1) have an attribute total:sum (2) have an attribute ( pagePath ) that starts with, but is not equal to, any element in the list mystrings .

Here is my list of strings, and my test data:

    mystrings = [u'/calculate-state-pension', u'/check-uk-visa']
    test_data = [
        {
            "pagePath": "/check-uk-visa",
            "total:sum": 2.0
        },
        {
            "pagePath": "/check-uk-visa/y",
            "total:sum": 3.0
        },
        {
            "pagePath": "/check-uk-visa/n",
            "total:sum": 4.0
        },
        {
            "pagePath": "/bank-holidays",
            "total:sum": 2.0
        },
        {
            "pagePath": "/check-uk-visa",
            "searchUniques:sum": 2.0
        }
    ]

So I would like to end up with this list:

    results = [
        {
            "pagePath": "/check-uk-visa",
            "total:sum": 2.0
        },
        {
            "pagePath": "/bank-holidays",
            "total:sum": 2.0
        },
        {
            "pagePath": "/check-uk-visa",
            "searchUniques:sum": 2.0
        }
    ]

This is my code:

    results = test_data[:]
    for r in results_copy:
        for s in mystrings:
            if 'total:sum' in r and r['pagePath'].startswith(s) \
                 and r['pagePath'] != s:
                results.remove(r)
    return results

But this doesn't seem to work. It removes the element with /check-uk-visa/y but not the one with /check-uk-visa/n .

What am I doing wrong? I think it's something to do with the deleting and the iterator - it looks like it's skipping elements.

Answer 1

You want any combination where "pagePath" value startswith a string in your string list but is not equal to the string.

for dic in test_data[:]:
    s = dic.get("pagePath","")
    if "total:sum" in dic and any(s.startswith(y) and s != y  for y in mystrings):
        test_data.remove(dic)

[{'total:sum': 2.0, 'pagePath': '/check-uk-visa'}, {'total:sum': 2.0, 'pagePath': '/bank-holidays'}, {'searchUniques:sum': 2.0, 'pagePath': '/check-uk-visa'}]

One caveat is if you have similar strings in your mystrings list where one might start with the same letters and not be equal but may be equal to another so in that case we can use a set for 0(1) lookups and use in.

mystrings = {u'/calculate-state-pension', u'/check-uk-visa'}

for dic in test_data[:]:
    s = dic.get("pagePath","")
    if "total:sum" in dic and any(s.startswith(y) for y in mystrings)and s not in mystrings:
        test_data.remove(dic)
print(test_data)

Answer 2

The easiest way to filter something like this is usually to use the filter function.

results_copy = filter(lambda r: ('total:sum' in r
                                 and any([r['pagePath'].startswith(s) 
                                                             for s in mystrings])
                                 and r['pagePath'] not in mystrings), 
                      results)

Alternatively, you could use a list comprehension. Sometimes it is easier to read when you want to do some processing in addition to filtering:

results_copy = [r for r in results if ('total:sum' in r
                                       and any([r['pagePath'].startswith(s) 
                                                             for s in mystrings])
                                       and r['pagePath'] not in mystrings)]