python asyncio从3.4迁移到3.5+

Question

Good evening everybody, I am trying to create internet robot and I met the problem while migrating my script from python 3.4 to 3.5 or 3.6+. It uses asyncio and works good on 3.4 python but when I start it with python3.5+ I got error: RuntimeError: Cannot run the event loop while another loop is running

here is the code scheme:

import multiprocessing as mp
import asyncio
import concurrent.futures
import aiohttp

def create_proccesses(separate_loop_creator, coro):
    proccesses = []
    for n in range(2):
        proc = mp.Process(target=separate_loop_creator, args=(coro,))
        proc.start()
        proccesses.append(proc)
    for p in proccesses:
        p.join()

def separate_loop_creator(coro):
    sep_loop = asyncio.new_event_loop()
    asyncio.set_event_loop(sep_loop)
    tasks = [asyncio.async(coro(sep_loop)) for _ in range(100)]
    try:
        sep_loop.run_until_complete(asyncio.wait(tasks))
        sep_loop.close()
    except Exception as err:
        print(err)
        for task in tasks:
            task.cancel()
        sep_loop.close()


@asyncio.coroutine
def manager(exe, loop):
    # some calculations and start coros in several processes
    loop.run_in_executor(
        exe,
        create_proccesses,
        separate_loop_creator,
        some_coro
    )

@asyncio.coroutine
def some_work_in_mainloop():
    while True:
        print('Some server dealing with connections here...')
        yield from asyncio.sleep(1)

@asyncio.coroutine
def some_coro(loop):
    with aiohttp.ClientSession(loop=loop) as session:
        response = yield from session.get('http://google.com')
        yield from asyncio.sleep(2)
        print(response.status)

if __name__ == '__main__':
    mainloop = asyncio.get_event_loop()
    executor = concurrent.futures.ProcessPoolExecutor(5)
    asyncio.async(some_work_in_mainloop())
    asyncio.async(manager(executor, mainloop))
    try:
        mainloop.run_forever()
    finally:
        mainloop.close()

The exception raise in separate_loop_creator() coroutine and it is RuntimeError: Cannot run the event loop while another loop is running . I think it is because of changing get_event_loop() mechnics, but I do not understand what is wrong with my code.

Here is what I want to do:

                       +--------------+
               +-------+other service |
    +----------+       +--------------+
    | mainloop |
    +----------+
               |     +------------+
               +-----+   executor |
                     +------+-----+
                            |
                     +------+--------+
                     |start proccess |
                     +---+-------+---+
+-----------------+      |       |      +---------------+
|start new loop   +------+       +------+ start new loop|
+--------+--------+                     +-------+-------+
         |                                      |
 +-------+-------+                       +------v-------+
 |   run coro    |                       | run coro     |
 +---------------+                       +--------------+

Here is trace which I get on python3.5.3:

Traceback (most recent call last):
  File "tst.py", line 21, in separate_loop_creator
    sep_loop.run_until_complete(asyncio.wait(tasks))
  File "/root/.pyenv/versions/3.5.3/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/root/.pyenv/versions/3.5.3/lib/python3.5/asyncio/base_events.py", line 411, in run_forever
    'Cannot run the event loop while another loop is running')
RuntimeError: Cannot run the event loop while another loop is running
Cannot run the event loop while another loop is running
Traceback (most recent call last):
  File "tst.py", line 21, in separate_loop_creator
    sep_loop.run_until_complete(asyncio.wait(tasks))
  File "/root/.pyenv/versions/3.5.3/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/root/.pyenv/versions/3.5.3/lib/python3.5/asyncio/base_events.py", line 411, in run_forever
    'Cannot run the event loop while another loop is running')
RuntimeError: Cannot run the event loop while another loop is running

Python 3.4.3 results:

...
200
Some server dealing with connections here...
200
200
Some server dealing with connections here...
200
200
Some server dealing with connections here...
200
...

Answer 1

This is actually a bug in asyncio in CPython 3.6.0. There's a PR to fix this, so that 3.6.1 works as expected.

As a workaround you can add the following piece of code in your project:

import sys

if sys.version_info[:3] == (3, 6, 0):
    import asyncio.events as _ae
    import os as _os

    _ae._RunningLoop._pid = None

    def _get_running_loop():
        if _ae._running_loop._pid == _os.getpid():
            return _ae._running_loop._loop

    def _set_running_loop(loop):
        _ae._running_loop._pid = _os.getpid()
        _ae._running_loop._loop = loop

    _ae._get_running_loop = _get_running_loop
    _ae._set_running_loop = _set_running_loop

Answer 2

The best solution, if possible, try removing multiprocessing altogether from your program, and use only one event loop (Optionally using ProcessPoolExecutor for isolated CPU intensive tasks).

As of 2017-03-02, there is an open python bug for this issue, effecting non-windows platforms: https://bugs.python.org/issue22087 .

Here is a shorter program to trigger the same problem:

import asyncio
import multiprocessing as mp


def create_another_loop():
    loop = asyncio.new_event_loop()
    loop.run_forever()


async def create_process():
    proc = mp.Process(target=create_another_loop)
    proc.start()
    proc.join()


if __name__ == '__main__':
    main_loop = asyncio.get_event_loop()
    main_loop.run_until_complete(create_process())
    main_loop.close()

A hackish workaround (Caution! Use at your own risk!) inspired from the fix suggested here https://github.com/python/asyncio/pull/497 is to add this code to a newly created Process :

if asyncio.events._running_loop:
    asyncio.events._running_loop._loop = None

Example:

import asyncio
import multiprocessing as mp
import time
from concurrent.futures.process import ProcessPoolExecutor


async def clock(label, n=5, sleep=1):
    print(label, "start")
    for i in range(n):
        await asyncio.sleep(sleep)
        print(label, i + 1)
    print(label, "end")
    return label


def create_another_loop():
    # HACK START
    if asyncio.events._running_loop:
        asyncio.events._running_loop._loop = None
    # HACK END

    loop = asyncio.new_event_loop()
    loop.run_until_complete(clock("sub"))
    loop.close()


def create_process():
    time.sleep(2)
    proc = mp.Process(target=create_another_loop)
    proc.start()
    proc.join()
    return "ok"


async def create_process_in_pool():
    return await main_loop.run_in_executor(ProcessPoolExecutor(), create_process)


if __name__ == '__main__':
    main_loop = asyncio.get_event_loop()
    tasks = (
        clock("main"),
        create_process_in_pool(),
    )
    print(main_loop.run_until_complete(asyncio.gather(*tasks)))
    main_loop.close()

Other possible workarounds: Create the processes before starting the loop or use asyncio.create_subprocess_exec which even allows to communicate with the subprocess via a stream .