Adding items to a list using threads in Python 2.7

Question

I've been doing some reading and couldn't find a best practice for my problem in Python 2.7, and it goes like that:

I've a list of items, I want to iterate through all of them and call to some function. This function will do some work with those items and return a new item instead. Each returned value from the function will be appended into a new list of items. Everything is nice and simple, till I try to do this using threads.

I wanna do the same job, but now each function call will open a new thread (max 20 threads) that will do the function's work and return the needed information back to be appended to the list.

newList = []

for item in myList:
    info = getInfo(item)
    newList.append(info)

return newList

From what I read, lists are thread-safe, so I thought maybe to use Queue and Threading modules then put all current items into a queue, do the job on each while passing the newList and appending inside every thread... Any ideas?

Answer 1

It's wisest to assume that operations aren't thread-safe unless you know beyond all doubt that they are.

The simplest way to ensure atomicity of any operation is to acquire a lock before performing the operation and only release it once the operation is performed. Nowadays the with statement makes that very simple, since lock objects are context managers.

Suppose your list is module global my_list then this is easily accomplished as follows.

my_lock = threading.Lock()
my_list = []
    :
    :
    :
def my_list_append(o):
    with my_lock:
        my_list.append(o)

Also note that the same lock must be used for any other modification of the list (like popping an item off it, for example). Simple access of elements by indexing in other threads should not be a problem.

Another possible method is to have a thread perform the operation and send items to be appended to the list via a Queue.queue object passed as an argument to the thread. This assumes only that thread ever appends anything to the list. Since the locking solution is so simple, I wouldn't consider this unless you have to.