Python List of list, remove duplicates

Question

Trying to remove duplicates in list of list and print same without duplicates.

Original List

a = [['country',['America_1','America_2','America_3','America_4','England_5','England_6'],['apple_1_more','orange_1_more']],['country',['Brazil_2','Brazil_3','Brazil_1','Brazil_4','Mexico_1','Mexico_3','Mexico_2'],['grapes_1_less','banana_1_more']]]

looking for output:

[['country', ['America', 'England'], ['orange_more', 'apple_more']], ['country', ['Mexico', 'Brazil'], ['grapes_less', 'banana_more']]]

but getting:

[['country', ['America', 'England'], ['orange_more', 'apple_more']], ['country', ['America', 'England', 'Mexico', 'Brazil'], ['orange_more', 'grapes_less', 'banana_more', 'apple_more']]]

code::

 a = [['country',['America_1','America_2','America_3','America_4','England_5','England_6'],['apple_1_more','orange_1_more']],['country',['Brazil_2','Brazil_3','Brazil_1','Brazil_4','Mexico_1','Mexico_3','Mexico_2'],['grapes_1_less','banana_1_more']]]
aa ={}
aaa=[]
aaaa=[]
aaaaa=[]
for i in a:
    for j in i[1]:
        j=j.split('_',1)[0]
        aaa.append(j)
    for k in i[2]:
        k=k.split('_',2)[0]+'_'+k.split('_',2)[2]
        aaaa.append(k)
    aa['country'] = [i[0],list(set(aaa)),list(set(aaaa))]
    aaaaa.append(aa['country'])
print (aaaaa)

Answer 1

Using a list comprehension, converting the second item in each sublist to and from a set() :

a = [['country',['America','America','America','America','England','England']],['country',['Brazil','Brazil','Brazil','Brazil','Mexico','Mexico','Mexico']]]

a = [[i, list(set(j))] for i, j in a]
print(a)

Output:

[['country', ['England', 'America']], ['country', ['Brazil', 'Mexico']]]

This may not preserve the order of the inner list, as sets are unordered, so you may need to account for this.

Answer 2

You can try this approach :

a = [['country',['America','America','America','America','England','England']],['country',['Brazil','Brazil','Brazil','Brazil','Mexico','Mexico','Mexico']]]



print(list(map(lambda x:[x[0],list(set(x[1:][0]))],a)))

output:

[['country', ['England', 'America']], ['country', ['Mexico', 'Brazil']]]

Your variables names are very confusing , Still i tried new approach , you can try this:

a = [['country',['America_1','America_2','America_3','America_4','England_5','England_6'],['apple_1_more','orange_1_more']],['country',['Brazil_2','Brazil_3','Brazil_1','Brazil_4','Mexico_1','Mexico_3','Mexico_2'],['grapes_1_less','banana_1_more']]]


final_data=[]
for i in a:
    sub_data=[]

    for j in i[1:]:
        d = {}

        for m in j:
            data=m.split('_')[0]
            d[data]=data

        sub_data.append(list(d.keys()))
    final_data.append(['country',*sub_data])
print(final_data)

output:

[['country', ['America', 'England'], ['orange', 'apple']], ['country', ['Brazil', 'Mexico'], ['banana', 'grapes']]]

If your data format is always like this then you can try this:

update

a = [['country',['America_1','America_2','America_3','America_4','England_5','England_6'],['apple_1_more','orange_1_more']],['country',['Brazil_2','Brazil_3','Brazil_1','Brazil_4','Mexico_1','Mexico_3','Mexico_2'],['grapes_1_less','banana_1_more']]]


final_data=[]
for i in a:
    sub_data=[]
    sub_extra=[]

    for j in i[1:2]:
        sub_extra.append(i[2])
        d = {}

        for m in j:
            data=m.split('_')[0]
            d[data]=data

        sub_data.extend([list(d.keys()),*sub_extra])
    final_data.append(['country',*sub_data])
print(final_data)

output:

[['country', ['America', 'England'], ['apple_1_more', 'orange_1_more']], ['country', ['Mexico', 'Brazil'], ['grapes_1_less', 'banana_1_more']]]

Answer 3

Use this recursive function to remove duplicate item in multi level array:

def dup(input_):
    if isinstance(input_, list):
        try:
            input_ = list(set([i.split('_')[0] if not isinstance(i, list) else i for i in input_]))
        except TypeError:
            pass
        for child in input_:
            input_[input_.index(child)] = dup(child)

    return input_

Answer 4

This is how I would go about it.

country_list1 = [a[0[0]]]
country_list2 = [a[1[0]]]
duplicates = [country for country in country_list1 in country_list2]
non_duplicates = [country for country in country_list1 not in country_list2]

This will give you both the duplicated ones and non-duplicated This is considering case sensitiveness of the names in both