使用 Multiprocessing 或 Threading 或 Asyncio 进行多任务处理，具体取决于场景

Question

I have my codes ready for 1 at a time performance, I wanna upgrade it to something fancy, multi-tasking. I am seeking helps about what I can use to achieve my goal.

my codes performs in this order: parsing multi-pages, parsing multi-posts, parsing multi-images. I tried to do multi-pages with multi-processing with pool.map(), it came out with KeyError of Daemonic can't have children processes. My understanding of this multi-tasking procedure is that parsing pages are fast, parsing posts and images can be really long.

What if I do parsing posts and parsing images together on single page, can it be allowed?

Which modules should i use to do so? thread? multiprocessing? asyncio? I went through a lot lately, I am struggling with what I should use.

Answer 1

So off the top of my head you can look at 2 things.

1) Asyncio (be careful this example uses threading and is not thread safe specifically the function asyncio.gather)

import asyncio
for work in [1,2,3,4,5]:
    tasks.append(method_to_be_called(work))

results = await asyncio.gather(*tasks)

2) Asyncio + multiprocessing https://github.com/jreese/aiomultiprocess