如何在没有熊猫的情况下删除重复项？

Question

This is the data

row1| sbkjd nsdnak ABC 
row2| vknfe edcmmi ABC
row3| fjnfn msmsle XYZ
row4| sdkmm tuiepd XYZ
row5| adjck rulsdl LMN

I have already tried this using pandas and got help from stackoverflow. But, I want to be able to remove the duplicates without having to use the pandas library or any library in general. So, only one of the rows having "ABC" must be chosen, only one of the rows having "XYZ" must be chosen and the last row is unique, so, it should be chosen. How do I do this? So, my final output should contain this:

[ row1 or row2 + row3 or row4 + row5 ]

Answer 1

This should only select the unique rows from your original table. If there are two or more rows which share duplicate data, it will select the first row.

data = [["sbkjd", "nsdnak", "ABC"],
        ["vknfe", "edcmmi", "ABC"],
        ["fjnfn", "msmsle", "XYZ"],
        ["sdkmm", "tuiepd", "XYZ"],
        ["adjck", "rulsdl", "LMN"]]

def check_list_uniqueness(candidate_row, unique_rows):
    for element in candidate_row:
        for unique_row in unique_rows:
            if element in unique_row:
                return False
    return True

final_rows = []
for row in data:
    if check_list_uniqueness(row, final_rows):
        final_rows.append(row)

print(final_rows)

Answer 2

This Bash command would do (assuming your data is in a file called test , and that values of column 4 do not appear in other columns)

cut -d ' ' -f 4 test | tr '\n' ' ' | sed 's/\([a-zA-Z][a-zA-Z]*[ ]\)\1/\1/g' | tr ' ' '\n' | while read str; do grep -m 1 $str test; done

cut -d ' ' -f 4 test chooses the data in the fourth column
tr '\\n' ' ' turns the column into a row (translating new line character to a space)
sed 's/\\([a-zA-Z][a-zA-Z]*[ ]\\)\\1/\\1/g' deletes the repetitions
tr ' ' '\\n' turns the row of unique values to a column
while read str; do grep -m 1 $str test; done while read str; do grep -m 1 $str test; done reads the unique words and prints the first line from test that matches that word