使列表唯一（python）

Question

I have a matrix of around 3000 species classifications eg

Arthropoda/Hexapoda/Insecta/Coleoptera/Cerambycidae/Anaglyptus

each line is a sequence of taxonomic classifications. What I need to do is, sort the 3000 lines so each one is unique so that the file can be fed to a program that creates phylogenetic(evolutionary) trees.

I have tried to use a set but get an error as lists are not hashable objects, however it is important to keep each line together as the values in each column for each line are nested.

Whats the best way to ensure I only have unique values in the last column but keep the integrity of each row?

many thanks

Answer 1

As mentioned in the comments, tuples are hashable, even though lists aren't. So let's convert your rows to tuples!

# Create the Dataset
L = []
L.append(["Arthropoda", "Hexapoda", "Insecta", "Coleoptera", "Cerambycidae", "Anaglyptus"])
L.append(["Arthropoda", "Hexapoda", "Insecta", "Coleoptera", "Cerambycidae", "Aromia"])

# Instead of a list of lists, let's have a list of tuples!
L = [tuple(x) for x in L]

# Using a set, we can easily remove duplicates
L = set(L)

Answer 2

The python masters may be offended but this answer is worth a try

l = []
with open('file.txt', 'r') as fp:
    for i in fp.readlines():
        if i not in l:
            l.append(i)

with open('file2.txt', 'w') as fp:
    fp.writelines(l)