python dict.items() 线程安全吗？

Question

Python raises an exception if a dictionary changes its size during iteration using iteritems() .

I am hit by this problem since my program is multithreaded and there are cases that I need to iterate over the dict while another thread is adding keys into the dict .

Fortunately, I don't need the iteration to be very precise on every element in the dict . Therefore I am thinking to use items() instead of iteritems() to do the iteration. I think items() makes a static snapshot of the dict and I would work around the problem.

My question is: does items() raises an exception if the dict size is changing at the same time with items() execution?

thanks

Answer 1

This answer pertains to CPython 's implementation of Python 2 and Python 3 .

Since the items() method is implemented purely in C without releasing the GIL beforehand or during its execution, no other thread may acquire it and the underlying data structure remains unchanged while this method is executed.

Thus, the invocation of the items() method in Python 2 and in Python 3 is guaranteed to succeed.

However, while the operations performed on the returned object are thread-safe in Python 2 , they are not in Python 3 .

Python 2

The returned object is a copy of the dictionary's list of (key, value) pairs. As such, it is decoupled from the dictionary and operations performed on the dictionary can't affect the copy in hand.

Though, it's worth noting that the items themselves aren't copies of the original items so care should be taken when dealing with mutable items.

Python 3

The returned object is a view object. As such, certain operations performed on it might fail, as detailed in the official documentation :

Iterating views while adding or deleting entries in the dictionary may raise a RuntimeError or fail to iterate over all entries.

Answer 2

As the excellent comments noted:

1. This is not thread safe.

2. You should really use a lock when doing such things.

It is possible to see this in the CPython source code, dictobject.c :

If you look at the function

static PyObject *
dict_items(register PyDictObject *mp)

which is used for items , you can see that (after some clever pre-allocation for the results), it basically iterates over the array mp->ma_table (using a mask to see where there are entries).

Now if you look at the function

static int
dictresize(PyDictObject *mp, Py_ssize_t minused)

which is used when the table needs to be resized, then you can see that ma_table 's elements can be moved into a completely different buffer, and then it can be freed using PYMem_Free .

So, you have a very real risk of accessing garbage memory, if things are done concurrently without synchronization.