简体   繁体   中英

How to remove duplicates from a list of tuples but keeping the original order

I want to remove redundant tuples but preserve the order of appearance. I looked at similar questions. This question Find unique rows in numpy.array looked very promising but somehow it did not work for me.

I could use pandas as in this answer ( https://stackoverflow.com/a/14089586/566035 ) but I prefer not using pandas so that the py2exe generated executable will be small.

import numpy as np

data = [('a','z'), ('a','z'), ('a','z'), ('1','z'), ('e','z'), ('c','z')]

#What I want is:
    array([['a', 'z'],
           ['1', 'z'],
           ['e', 'z'],
           ['c', 'z']], 
          dtype='|S1')

#What I have tried:
# (1) numpy.unique, order not preserved
np.unique(data)

    array([['a', 'z'],
           ['c', 'z'],
           ['1', 'z'],
           ['e', 'z']], 
          dtype='|S1')

# (2) python set, order not preserved
set(data)

    set([('1', 'z'), ('a', 'z'), ('c', 'z'), ('e', 'z')])

# (3) answer here : https://stackoverflow.com/a/16973510/566035, order not preserved
a = np.array(data)
b = np.ascontiguousarray(a).view(np.dtype((np.void, a.dtype.itemsize * a.shape[1])))
_, idx = np.unique(b, return_index=True)

a[idx]

    array([['1', 'z'],
           ['a', 'z'],
           ['c', 'z'],
           ['e', 'z']], 
          dtype='|S1')

这在效率方面不是很好,但是可读代码非常简单,可以用于较小的列表:

sorted(set(data), key=data.index)

Oops! I found an answer myself...

seen = set()
np.array([x for x in data if x not in seen and not seen.add(x)])

# output
array([['a', 'z'],
       ['1', 'z'],
       ['e', 'z'],
       ['c', 'z']], 
      dtype='|S1')

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM