如何读取和处理 Python 中对于 memory 来说太大的文件？

Question

I have csv file that looks like:

1,2,0.2
1,3,0.4
2,1,0.5
2,3,0.8
3,1,0.1
3,2,0.6

First column correspond to user_a , second to user_b and third correspond to score . I want to find for every user_a , a user_b value that maximizes the score . For this example output should look like (output in form of dictionary preferable but not requred):

1 3 0.4
2 3 0.8
3 2 0.6

The problem is that file is very big (millions of rows) and I try to find way to do it without out of memory error. Because of environment setup I cannot use Pandas , Dask and other packages with dataframes.

I created code to read large file line by line:

def read_large_data(file_name, sep=","):
    for line in open(file_name, "r"):
        a, b, s = line.rstrip().split(sep)
        yield a, b, s

And code for finding max score:

def find_max_score(file_name, sep=","):
    result = {"User_A": [], "User_B": [], "Score": []}
    read_file_gen = read_large_data(file_name, sep)
    while True:
        try:
            a, b, s = next(read_file_gen)
            if a not in result["User_A"]:
                result["User_A"].append(a)
                result["User_B"].append(b)
                result["Score"].append(s)
            else:
                ind = result["User_A"].index(a)
                if s > result["Score"][ind]:
                    result["User_B"][ind] = b
                    result["Score"][ind] = s
        except StopIteration:
            break
    return result

I used the yield function to keep the memory needed for computation, but I still get an out-of-memory error. Any advice on how to reduce memory consumption would be highly appreciated.

Answer 1

In a comment you are saying "Yes, correct, the real data is ordered pairs and sorted" . So why can't you just do the following:

import csv
from itertools import groupby
from operator import itemgetter

def max_key(row): return float(row[2])

def find_max_score(file_name, sep=","):
    result = {"User_A": [], "User_B": [], "Score": []}
    with open(file_name, "r") as file:
        reader = csv.reader(file, delimiter=sep)
        for user_a, rows in groupby(reader, key=itemgetter(0)):
            _, user_b, score = max(rows, key=max_key)
            result["User_A"].append(user_a)
            result["User_B"].append(user_b)
            result["Score"].append(score)
    return result