Compare items in list with nested for-loop

Question

I have a list of URLs in an open CSV which I have ordered alphabetically, and now I would like to iterate through the list and check for duplicate URLs. In a second step, the duplicate should then be removed from the list, but I am currently stuck on the checking part which I have tried to solve with a nested for-loop as follows:

for i in short_urls:
    first_url = i
    for s in short_urls:
        second_url = s
    if i == s:
       print "duplicate"
    else:
       print "all good"

The print statements will obviously be replaced once the nested for-loop is working. Currently, the list contains a few duplicates, but my nested loop does not seem to work correctly as it does not recognise any of the duplicates.

My question is: are there better ways to do perform this exercise, and what is the problem with the current nested for-loop?

Many thanks :)

Answer 1

By construction, your method is faulty, even if you indent the if/else block correctly. For instance, imagine if you had [1, 2, 3] as short_urls for the sake of argument. The outer for loop will pick out 1 to compare to the list against. It will think it's finding a duplicate when in the inner for loop it encounters the first element, a 1 as well. Essentially, every element will be tagged as a duplicate and if you plan on removing duplicates, you'll end up with an empty list.

The better solution is to call set(short_urls) to get a set of your urls with the duplicates removed. If you want a list (as opposed to a set ) of urls with the duplicates removed, you can convert the set back into a list with list(set(short_urls)) .

In other words:

short_urls = ['google.com', 'twitter.com', 'google.com']
duplicates_removed_list = list(set(short_urls))
print duplicates_removed_list # Prints ['google.com', 'twitter.com']

Answer 2

if i == s:

is not inside the second for loop. You missed an indentation

for i in short_urls:
    first_url = i
    for s in short_urls:
        second_url = s
        if i == s:
           print "duplicate"
        else:
           print "all good"

EDIT: Also you are comparing every element of an array with every element of the same array. This means compare the element at position 0 with the element at postion 0, which is obviously the same. What you need to do is starting the second for at the position after that reached in the first for.