Asyncio和multiprocessing.Process-如何传递协程

Question

I've got a coroutine which is scraping some data. I'm trying to add multiprocessing to speed it up. I've come up with using multiprocesesing.Process. My coroutine is called fetch_students_pages

What I'm trying to do is to pass this coroutine as target to Process class:

    def fetch_students_concurrently(self):
        for _ in range(10):
            Process(target=self.fetch_students_pages).start()

but if fails with:

/usr/lib/python3.7/multiprocessing/process.py:99: RuntimeWarning: coroutine 'StudentPageParser.fetch_students_pages' was never awaited self._target(*self._args, **self._kwargs) RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Is there a way to await it or use another solution instead?

Answer 1

Consider using map() instead:

cores = 8

with Pool(cores) as p:
    p.map(self.fetch_students_pages, range(cores))