仅向python列表添加唯一项？

Question

I am running a python scraping script and am getting a list like this -

[u'UI/UX Designer\xa0\u2013 Creative Head ', u'UX Designer ', u'UI/UX Designer\xa0\u2013 Creative Head', u'UX Designer']

I wanted to add unique items only from the list so i used this -

profile_list = []
k = soup.body.findAll(text=re.compile("UX Designer"))
    for i in k:
        if i not in profile_list:
            profile_list.append(i)
    print profile_list

But it is not working, duplicate items still remain. I also tried to use set() as well but it is also not working here. What should i do here to add the unique items only?

Update - Thank you for the answers, i did a silly mistake here, two of the repeating words in the list have extra space here which should be removed. All the answers are correct so picked the oldest.

Answer 1

The first two strings in the list contains trailing spaces.

A string with a space and the other string without a space are different even though other characters are same:

>>> 'a' == 'a '
False

You need to strip them:

for i in k:
    i = i.strip()  # <----
    if i not in profile_list:
        profile_list.append(i)

---

UPDATE If the order of list items is not important, you can use set :

profile_list = list(set(s.strip() for s in k))  # Using `set` with generator expression

profile_list = list({s.strip() for s in k})  # Using set comprehension

Answer 2

There is a trailing space at the end of your strings, you should strip the excess whitespace. Use set or list comprehensions to make your code Pythonic. If you want the elements to be unique I also suggest using a set:

>>> st = [u'UI/UX Designer\xa0\u2013 Creative Head ', u'UX Designer ', u'UI/UX Designer\xa0\u2013 Creative Head', u'UX Designer']
>>> uniques = {elem.strip() for elem in st}
>>> uniques
set([u'UX Designer', u'UI/UX Designer\xa0\u2013 Creative Head'])

Answer 3

Looking at the output, the code you are using is actually working. The problem is, there is an extra space in the text:

[u'UI/UX Designer\xa0\u2013 Creative Head ', # Note the space here
u'UX Designer ', # and here
u'UI/UX Designer\xa0\u2013 Creative Head',
u'UX Designer'
]

All you need to do is strip() them:

profile_list = []
k = soup.body.findAll(text=re.compile("UX Designer"))
for i in k:
    if i.strip() not in profile_list:
        profile_list.append(i.strip())
print profile_list

Another way, as mentioned by @edwinskl is to make it a set() from the beginning:

profile_list = set()
k = soup.body.findAll(text=re.compile("UX Designer"))
for i in k:
    if i.strip() not in profile_list:
        profile_list.add(i.strip())
print profile_list

Or another way (which I thought of when I first looked at your question) is to make it a set afterwards:

profile_list = []
k = soup.body.findAll(text=re.compile("UX Designer"))
for i in k:
    profile_list.append(i.strip())
list(set(profile_list))
print profile_list

though it isn't as good as the two above.