如何使用Python从输出中删除重复项？

Question

facing a issue here:

Following example:

for item in g_data:
        Header = item.find_all("div", {"class": "InnprodInfos"})
        print(Header[0].contents[0].text.strip())

Output:

DMZ 3rd Tunnel - Korean Demilitarized Zone Day Tour from Seoul
Panmunjeom Day Tour
Seoul City Half Day Private Tour
The Soul of Seoul - Small Group Tour
Seoul Helicopter Tour
Seoul City Full Day Tour
Seoul City Half Day Tour
The Street Museum in the Urban Core - Small Group Tour
Korean Folk Village Day Tour
DMZ 3rd Tunnel - Korean Demilitarized Zone Day Tour from Seoul
Panmunjeom Day Tour
Seoul City Half Day Private Tour
The Soul of Seoul - Small Group Tour
Seoul Helicopter Tour
Seoul City Full Day Tour
Seoul City Half Day Tour
The Street Museum in the Urban Core - Small Group Tour
Korean Folk Village Day Tour

As you can see above, it gives me the output twice. Hence, only the second duplicates should be removed.

The result should look like:

DMZ 3rd Tunnel - Korean Demilitarized Zone Day Tour from Seoul
Panmunjeom Day Tour
Seoul City Half Day Private Tour
The Soul of Seoul - Small Group Tour
Seoul Helicopter Tour
Seoul City Full Day Tour
Seoul City Half Day Tour
The Street Museum in the Urban Core - Small Group Tour
Korean Folk Village Day Tour

Can anyone provide me feedback how to delete the duplicates? Any feedback is appreciated.

Answer 1

You can use list or set (if order doesn't matter):

Using list:

result = []
for item in g_data:
    header = item.find_all("div", {"class": "InnprodInfos"})
    item = header[0].contents[0].text.strip()
    if item not in result:
        result.append(item)

print '\n'.join(result)

Using set:

result = set()
for item in g_data:
    header = item.find_all("div", {"class": "InnprodInfos"})
    result.add(header[0].contents[0].text.strip())

print '\n'.join(result)

Answer 2

You should store the output in a set to verify if it has been "printed" already. After that you print out the elements of the set.

g_data = ["foo", "bar", "foo"]
g_unique = set()
for item in g_data:
        g_unique.add(item) # ensures the element will only be copied if not already in the set

for item in g_unique:
    print(item) # {'foo', 'bar'}

Answer 3

You can use a set to keep track of which items you have printed. This preserves the original order

already_printed = set()
for item in g_data:
    header = item.find_all("div", {"class": "InnprodInfos"})
    item = header[0].contents[0].text.strip()
    if item not in already_printed:
        print(item)
        already_printed.add(item)

Answer 4

There is a simple way to do this using list comprehension :)

s = set()
[s.add(text) for d_text in Header[0].contents[0].text.strip().split('\n')]
print('\n'.join([text for text in s]))