在单个列表中删除相似（但不相同）的字符串

Question

I have a list of strings that look like:

my_list = ['https://www.google.com/', 'http://www.google.com/', 
           'https://www.google.com',  'http://www.google.com']

As you can see they are not the same but they all look very similar.

I also have a function which is:

from fuzzywuzzy import fuzz

def similar(a, b):
    return fuzz.ratio(a,b)

I want to use this functions and say something like:

for a,b in my_list:
    print (a,b)
    if similar(a,b) > 0.95:
        my_list.remove(b)

So I'm trying to remove similar looking strings from a list if they are above a certain similarity ratio. I want to do this so that in this list I would end up with just the first url, in this case my_list would end up being:

my_list = ['https://www.google.com/']

Answer 1

After doing some googling, I found fuzzywuzzy has an inbuilt function which is pretty great.

from fuzzywuzzy.process import dedupe

deduped_list = list(dedupe(my_list, threshold=97, scorer=fuzz.ratio))

Answer 2

In general you should never use list.remove() within an iteration loop, because the list iterator will get confused when you remove an item from the same list you are iterating over.

And because you always want to keep the first item you can exclude it from the loop:

idx = 1
while idx < len(my_list):
    if similar(my_list[idx - 1], my_list[idx]) > 0.95:
        my_list.remove(my_list[idx])

print(my_list)

alternative solution with list comprehension

first_item = my_list[0]
my_list = [first_item] + [item for item in my_list[1:] if similar(first_item, item) <= 0.95]

print(my_list)