python: if key not in dictionary add the key. preventing race conditions with multiple threads

Question

i am programming a telegram bot where more thread handles every update async. i hard code an empty dictionary and while the bot is running, multiple threads write and read from it.

The most common operation is:

if key not in dict:
    dict[key] = value
else:
    dict[key].append(another_value)

the problem is that while a thread check if key is or not in dict, and maybe it is not, another thread wrote the key in it.

so basically i have to get rid of this racing condition.

i need a fast solution. searching on the web i found answers talking about threading.Lock() . Those answers are about 10 years old. i wonder if nowadays it's still a good solution or maybe there is some more new library

Answer 1

You can use a defaultdict or, if you want to use a regular dict, the setdefault() method.

Using defaultdict :

from collections import defaultdict
mydict = defaultdict(list)    # new key is initialized with an empty list
mydict[key].append(value)     # so it is always safe to append

Using setdefault() :

mydict = {}
mydict.setdefault(key, []).append(value)

Either should be thread-safe, IIRC, even if they don't look that way: the method implementations are in C and therefore can't be interrupted by Python code. So it is never possible to, for example, have two setdefault() methods executing at literally the same time and causing a race condition.

Generally defaultdict is preferred because it is more performant, makes empty lists only when needed, and the resulting code is simpler. The setdefault() version could be better if some keys in your dictionary might have non-list values.

Answer 2

Python's dicts are safe for single operations, but multiple operations are not. What you're currently doing is multiple operations. Locks are fine, and generally the standard way to handle this problem.

Take a look here to see some of the details regarding Python's thread-safety.

http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm