How to remove many elements in a very big dict?

Question

I have a very large dict , and I want to del many elements from it. Perhaps I should do this:

new_dict = { key:big_dict[key] for key in big_dict if check(big_dict[key] }

However, I don't have enough memory to keep both old_dict and new_dict in RAM. Is there any way to deal?

Add:

I can't del elements one by one. I need to do a test for the values to find which elements I want to del.

I also can't del elements in a for loop like:

for key in dic: 
    if test(dic(key)): 
        del dic[key]

It case a error... Can't change len(dic) in the loop...

My God... I even can't make a set to remember keys to del, there are too much keys...

I see, if dict class don't have a function to do this, perhaps the only way to do this is to bug a new computer...

Answer 1

You could try iterating through the dictionary and deleting the element that you do not require by

del big_dict[key]

This way you wouldn't be making copies of the dictionary.

Answer 2

Here are some options:

Make a new 'dict' on disk, for which pickle and shelve may be helpful.
Iterate through and build up a list of keys until it reaches a certain size, delete those, and then repeat the iteration again, allowing you to make a bigger list each time.
Store the keys to delete in terms of their index in .keys() , which can be more memory efficient. This is OK as long as the dictionary is not modified between calls to .keys() . If about half of the elements are to be deleted, do this with a binary sequeunce (1 = delete, 0 = keep). If a vast majority of elements are to be deleted (or not deleted) store the appropriate keys as integers in a list.

Answer 3

You can use

big_dict.pop("key", None)