如何从python中的txt文件中删除带有重复子字符串的行？

Question

i am trying to remove lines from a .txt file that contains duplicated substrings. lets say i have lines like this:

aaaaaa, something.... 
bbbbbb, something differet.. 
cccccc, some other text.. 
cccccc, again different text.. 
dddddd, again some other text..
eeeeee, some other text... 
etc..

i want to filter out all the lines that start with the same substring (first N chars), so that there will be only one (the first one) line starting with it. these i want to copy to a new txt file.

so in the example above the first three lines would be copied, the fourth would be skipped and the rest would be copied.

i want to copy all the lines, not only the substring that i am checking

this is what i have written based on what i have found

lines_seen = set()
outfile = open(outfile, "w")

for line in open(infile, "r"):
    string_to_compare = line[0:N] #save the substring into a variable
    if line.startswith(string_to_compare) not in lines_seen:
        outfile.write(line)
        lines_seen.add(line)
outfile.close()

this code above actually copies all the lines from the outfile into infile, so no filtering is done.

can anyone tell me where is the mistake or how to make it work, please?

Answer 1

如果只对前 60 个字符感兴趣，你应该只在你的集合中存储这个切片（ lines_see.add(string_to_compare) ）并且你的检查应该更改为if string_to_compare not in lines_seen: 。