python code to remove duplicates or substrings of element from list

Question

I have the following list as input:

['temp/date=20-07-2019/', 'temp/date=21-07-2019/', 'temp/date=22-07-2019/', 'temp/date=22-07-2019/temp=22-07-2019/']

In the output I want to exclude 'temp/date=22-07-2019/' since its a part of 'temp/date=22-07-2019/temp=22-07-2019/'. Hence the output should be:

['temp/date=20-07-2019/', 'temp/date=21-07-2019/', 'temp/date=22-07-2019/temp=22-07-2019/']

I have tried several ways but was not able to achieve this. Please suggest. Thanks

Answer 1

You can use any with a list comprehension:

r = ['temp/date=20-07-2019/', 'temp/date=21-07-2019/', 'temp/date=22-07-2019/', 'temp/date=22-07-2019/temp=22-07-2019/']
result = [i for i in r if not any(i in c and len(c) > len(i) for c in r)]

Output:

['temp/date=20-07-2019/', 'temp/date=21-07-2019/', 'temp/date=22-07-2019/temp=22-07-2019/']

Answer 2

In case your items have specific format ( temp/date=DD-MM-YY/ ):

d = {}
lst = ['temp/date=20-07-2019/', 'temp/date=21-07-2019/',
       'temp/date=22-07-2019/', 'temp/date=22-07-2019/temp=22-07-2019/']

for s in lst:
    k = s[:21]
    if k not in d or len(s) > len(d[k]):
        d[k] = s

print(list(d.values()))

The output:

['temp/date=20-07-2019/', 'temp/date=21-07-2019/', 'temp/date=22-07-2019/temp=22-07-2019/']

Answer 3

You can use a dictionary:

lst = ['temp/date=20-07-2019/', 'temp/date=21-07-2019/', 'temp/date=22-07-2019/', 'temp/date=22-07-2019/temp=22-07-2019/']
dct = {re.match(r'(temp/.+?)/.*', i).group(1): i for i in sorted(lst, key=len)}
# {'temp/date=20-07-2019': 'temp/date=20-07-2019/', 'temp/date=21-07-2019': 'temp/date=21-07-2019/', 'temp/date=22-07-2019': 'temp/date=22-07-2019/temp=22-07-2019/'}

print(list(dct.values()))
# ['temp/date=20-07-2019/', 'temp/date=21-07-2019/', 'temp/date=22-07-2019/temp=22-07-2019/']

Answer 4

This solution is also taking care of identical duplicates creating a set:

example_data = ['temp/date=20-07-2019/', 'temp/date=21-07-2019/', 'temp/date=22-07-2019/', 'temp/date=22-07-2019/temp=22-07-2019/']
# Creating a set we get rid of all identical entries
unique_data = set(example_data)
result = []

# Here we cycle through unique_data adding to result list all the long strings
# since we want to keep all the long entries
[result.append(item) for item in unique_data if len(item) > 21]

# Then we cycle again and take care of adding to result all the short strings that
# are not already contained in result items
for item in unique_data:
    if len(item) == 21:
        for element in result:
            if item != element[:21]:
                result.append(item)
            break

# I am not sure you need to sort by date but this can be easily achieved with sorted
print(sorted(result))

Answer 5

I would suggest using a List Comprehensive approach because your data is already in a list. Here is some basic information related to using list comprehensive and other comprehensive techniques. In the code example below, I added a couple of extra data items to your original list.

from pprint import pprint

input_data = ['temp/date=20-07-2019/',
            'temp/date=20-07-2019/',
            'temp/date=21-07-2019/',
            'temp/date=22-07-2019/',
            'temp/date=23-07-2019/',
            'temp/date=21-07-2019/temp=22-07-2019/',
            'temp/date=22-07-2019/temp=22-07-2019/']

##################################################################################
# This List Comprehensive does several functions:
# 1. Creates a Set() of unique strings that are contained in the input_data list 
# 2. Sorts the Set() which will produce an ordered output
# 3. Uses the not any(iterable)
##################################################################################
result = [x for x in sorted(set(input_data)) if not any(x in y and len(y) > len(x) for y in input_data)]

# The output removed the duplicate string 'temp/date=20-07-2019/' it
# also removed the strings 'temp/date=21-07-2019/' and 'temp/date=22-07-2019/'
# which were contained in other strings in the list.
pprint (result)
['temp/date=20-07-2019/', 
 'temp/date=21-07-2019/temp=22-07-2019/', 
 'temp/date=22-07-2019/temp=22-07-2019/', 
 'temp/date=23-07-2019/']