大文件和python

Question

I have a large CSV file(300+ GB) with data (time, OS, ID). What should I do to count the IDs that occur more than 1 time in this file? Which algorithm will not overflow?

Answer 1

Simple for loop and reading by the line should do it

result_set = set()
with open(filename, "r") as input_file:
    for line in input_file:
        # Expected format is smth, smth, important
        splitted_list = line.split(",")
        result_set.add(splitted_list[-1].strip())
print(result_set)
# If file is
# ---
# random, random, important1
# random, random, important2
# ---
# prints:
# {'important2', 'important1'}

Unlike readlines(), this does not load the whole file. It will take it's sweet time but it won't crash.