在Python中如何确保列表中的项目被成功处理

Question

I try this:

for k in  keywords_list:
    google_add = random.choice(google_adds_list)
    url = make_up_url(google_add, k, False)
    if scrape_keyword_count(k, useragent_list, url, result_dir):
        keyword_count = scrape_keyword_count(k, useragent_list, url, result_dir)
        all_keyword_count.append(keyword_count)
        print '%s Finish. Removeing it from the list' % k
        keywords_list.remove(k)
    else:
        print "%s may run into problem, removing it from list" % google_add
        google_adds_list.remove(google_add)
        with open(google_adds, 'w') as f:
            f.write('\n'.join(google_adds_list))

I set up many reverse proxy server for google. the server list is google_add_list I mean to search all the item in the list with the add i provide and get the result If google block me, the scrape_keyword_count() will return None. then i And I need to change to another add to do the search. but the script i wrote will skip the keyword no matter the scrape_keyword_count() success or not

I know removing an item within the for loop is dangerous i will improve this part later

Answer 1

The problem here is that you're modifying the list while iterating over it.

Use "for i in the_list[:]" instead. This will iterate over a copy of the list, fixing your "skipping" (missed elements) issue.

Answer 2

Perhaps:

new_list = []
for i in the_list:
    if do_something_with(i):
       continue
    do_something_else(i)
    new_list.append(i)

If do_something_with(i) succeeded then continue with the next item otherwise do_something_else(i) .

You can't mutate a list while iterating over it. If the filtered list is needed, rather than removing elements from the old one produce a new one.

Answer 3

The for loop will use each item once... Your code looks fine... But I think you may not be returning the correct value in do_something_with

Try this:

for i in the_list:
    value = do_something_with(i)
    print bool(value)
    if value:
        the_list.remove(i)
    <etc> <etc>

I think you may always be returning True from do_something_with