在python中管理字典内存大小

Question

I have a program which imports a text file through standard input and aggregates the lines into a dictionary. However the input file is very large (1Tb order) and I wont have enough space to store the whole dictionary in memory (running on 64Gb ram machine). Currently Iv got a very simple clause which outputs the dictionary once it has reached a certain length (in this case 100) and clears the memory. The output can then be aggregated at later point.

So i want to: output the dictionary once memory is full. what is the best way of managing this? Is there a function which gives me the current memory usage? Is this costly to keep on checking? Am I using the right tactic?

import sys
X_dic = dict()

# Used to print the dictionary in required format
def print_dic(dic):
    for key, value in dic.iteritems():
        print "{0}\t{1}".format(key, value)

for line in sys.stdin:
    value, key = line.strip().split(",")      

    if (not key in X_dic):
        X_dic[key] = []                            

    X_dic[key].append(value)

    # Limit size of dic.
    if( len(X_dic) == 100):
        print_dic(X_dic)              # Print and clear dictionary
        X_dic = dict()


# Now output
print_dic(X_dic)

Answer 1

The module resource provides some information on how much resources (memory, etc.) you are using. See here for a nice little usage.

On a Linux system (I don't know where you are) you can watch the contents of the file /proc/meminfo . As part of the proc file system it is updated automatically.

But I object to the whole strategy of monitoring the memory and using it up as much as possible, actually. I'd rather propose to dump the dictionary regularly (after 1M entries have been added or such). It probably will speed up your program to keep the dict smaller than possible; also it presumably will have advantages for later processing if all dumps are of similar size. If you dump a huge dict which fit into your whole memory when nothing else was using memory, then you later will have trouble re-reading that dict if something else is currently using some of your memory. So then you would have to create a situation in which nothing else is using memory (eg reboot or similar). Not very convenient.