如何通过合并 function 在 Python 中实现多线程/多处理

Question

The dictionary and code is below which is working

• The question is regarding mainly on multithreading, I know that below code can be easily rewrite to easy way. But the ask is on multithreading, I just created one working example to test
• total(todos) is the main function
• user_count, title_count, complete_count are independent of each other
• I need to implement multithreading/multiprocessing has to implement
• def total(todos): is the place where need to do multithreading

todos = [{'userId': 1, 'id': 1, 'title': 'A', 'completed': False},
     {'userId': 1, 'id': 2, 'title': 'B ', 'completed': False},
     {'userId': 1, 'id': 1, 'title': 'C', 'completed': False},
     {'userId': 1, 'id': 2, 'title': 'A', 'completed': True},
     {'userId': 2, 'id': 1,'title': 'B', 'completed': False}]
def total(todos):
    ###### Multithreading need to implement ##########
    user_count = userid(todos)
    title_count = title(todos)
    complete_count = completed(todos)
    search_count_all = {**user_count, **title_count, **complete_count}
    return search_count_all
def userid(todos):    
    for d in todos:
        for l, m in d.items():  
            super_dict.setdefault(l, []).append(m)
    d = {k:len(set(v)) for k,v in super_dict.items()}
    return {"userid":d['userId']}
def title(todos):    
    for d in todos:
        for l, m in d.items():  
            super_dict.setdefault(l, []).append(m)
    d = {k:len(set(v)) for k,v in super_dict.items()}
    return {"title":d['title']}
def completed(todos):    
    for d in todos:
        for l, m in d.items():  
            super_dict.setdefault(l, []).append(m)
    d = {k:len(set(v)) for k,v in super_dict.items()}
    return {"completed":d['completed']}

total(todos)

Current output and expected output

{'userid': 2, 'title': 4, 'completed': 2}

can we do multprocessing also

from joblib import Parallel, delayed

Answer 1

As mandulaj answer specifies, you can use the library as follows. I simplified the example, so it's easier to reason.

I wrote 3 different functions, where each just multiply the input value by a different number. Then using the ThreadPoolExecutor I submitted their execution, concurrently.

You then need to collect the results of the each execution. This is done with the as_completed method from the library.

from concurrent.futures import ThreadPoolExecutor, as_completed


def func1(i):
    return i


def func2(i):
    return i*2


def func3(i):
    return i*3


funcs_to_execute = [func1, func2, func3]
arg = 1
res = []

with ThreadPoolExecutor(max_workers=2) as executor:
    futures = {executor.submit(f, arg) for f in funcs_to_execute}

    for fut in as_completed(futures):
        res.append(fut.result())

print(res)

>[2, 1, 3]

To apply it for your use-case, the function you submit is userid, title, completed with the arg todo . The results of all of them resides in the res list container.

However, there are few issues with your code when you enable concurrency. It's currently working for you, because it runs sequentially. The superdict is shared, and when accessed concurrently, is not guaranteed. You can fix this by creating a new list in every function, return it as a result, and at the end, aggregate. So something like this

def userid(todos):
    res = list()
    for d in todos:
        for l, m in d.items():
            if l == 'userid':
                res.append(m)

    return {"userid": res}

and the merge the results from futures (which are, dictionaries) into a single superdict .

Answer 2

Check out pythons concurrent.futures library. You can run your function as a thread with

import concurrent.futures

with concurrent.futures.ThreadPoolExecutor() as executor:
    future1 = executor.submit(userid, todos)
    future2 = executor.submit(title, todos)
    # ...

    user_count = future1.result()
    title_count = future2.result()

Answer 3

As I mentioned in my comment to your question, you really should be using multiprocessing rather than multithreading since your tasks are CPU-intensive. The problem then becomes sharing a global super_dict dictionary since, unlike with multithreading, each subprocess has its own memory space and super_dict must now be located in shared memory. The way to go about initializing this is to use a dict instance returned by the multiprocessing.SyncManager and to then initialize each subprocess with a proxy to this managed dictionary when the process pool is created. This method will work for both Windows and Unix-based platforms. This version of the dictionary has some limitations. It does not seem to support the setdefault method and append method the way you would want it to in that the proxy does not seem to realize that the dictionary has actually been updated. So, a few adjustments have to be made.

But there is a great inefficiency in your code. Each of your functions userId , title and completed perform the same redundant processing of the todos list against the super_dict dictionary adding multiple occurrences of user ids, titles, etc. This doesn't ultimately matter because you are counting the number of unique values. But it is wasteful of CPU cycles to do this redundantly and in the multiprocessing scenario you now have the problem of handling the parallel updating of the dictionary that buys you nothing. The following code therefore just initializes super_dict in the main process. And your functions userId , title and completed are doing more calculations than they need to, which can be further simplified (see below).

When one is done improving the efficiency of the code, it does not pay to actually even bother with multiprocessing for this particular example since the overhead of creating the pool and proxy dictionary is greater than any savings from executing these three trivial functions in parallel. But this, at least, shows you the technique for a real example.

from concurrent.futures import ProcessPoolExecutor
from multiprocessing import Manager


def init_pool(d):
    global super_dict
    super_dict = d

def total(todos):
    # Call to Manager() returns a SyncManger instance:
    with Manager() as manager:
        d = manager.dict()
        init_dict(d)
        with ProcessPoolExecutor(max_workers=3, initializer=init_pool, initargs=(d,)) as executor:
            f1 = executor.submit(userid)
            f2 = executor.submit(title)
            f3 = executor.submit(completed)
            user_count = f1.result()
            title_count = f2.result()
            complete_count = f3.result()
            search_count_all = {**user_count, **title_count, **complete_count}
            return search_count_all

def init_dict(the_dict):
    for d in todos:
        for l, m in d.items():
            # Cannot use setdefault:
            if l in the_dict:
                # Use this instead of append:
                the_dict[l] += [m]
            else:
                the_dict[l] = [m]

def userid():
    return {'userId': len({v for v in super_dict['userId']})}

def title():
    return {'title': len({v for v in super_dict['title']})}

def completed():
    return {'completed': len({v for v in super_dict['completed']})}

if __name__ == '__main__':
    todos = [{'userId': 1, 'id': 1, 'title': 'A', 'completed': False},
         {'userId': 1, 'id': 2, 'title': 'B ', 'completed': False},
         {'userId': 1, 'id': 1, 'title': 'C', 'completed': False},
         {'userId': 1, 'id': 2, 'title': 'A', 'completed': True},
         {'userId': 2, 'id': 1,'title': 'B', 'completed': False}]
    print(total(todos))