Python 3.7.2 在 asyncio.run() 关闭期间从未处理的异常构建期间测试失败

Question

I'm trying to build Python 3.7.2 in a Ubuntu 18.04 Docker container with the build being optimized ( --enable-optimizations and --with-lto ). During this build, the tests (specifically it looks like test_sys_settrace ) fail with

unhandled exception during asyncio.run() shutdown
task: <Task finished coro=<<async_generator_athrow without __name__>()> exception=RuntimeError("can't send non-None value to a just-started coroutine")>
RuntimeError: can't send non-None value to a just-started coroutine
unhandled exception during asyncio.run() shutdown
task: <Task finished coro=<<async_generator_athrow without __name__>()> exception=RuntimeError("can't send non-None value to a just-started coroutine")>
RuntimeError: can't send non-None value to a just-started coroutine

It isn't very clear to me why this is happening, and searching on unhandled exception during asyncio.run() shutdown points to issues with the code, which I don't have control over here. I should also note that this works fine with no errors if the build is for Python 3.6.8.

What follows it the relevant parts of my Dockerfile

FROM ubuntu:bionic

USER root
WORKDIR /root

SHELL [ "/bin/bash", "-c" ]

ARG PYTHON_VERSION_TAG=3.7.2

# Existing lsb_release causes issues with modern installations of Python3
# https://github.com/pypa/pip/issues/4924#issuecomment-435825490
# Set (temporarily) DEBIAN_FRONTEND to avoid interacting with tzdata
RUN apt-get -qq -y update && \
    apt-get -qq -y upgrade && \
    DEBIAN_FRONTEND=noninteractive apt-get -qq -y install \
        gcc \
        g++ \
        zlibc \
        zlib1g-dev \
        libssl-dev \
        libbz2-dev \
        libsqlite3-dev \
        libncurses5-dev \
        libgdbm-dev \
        libgdbm-compat-dev \
        liblzma-dev \
        libreadline-dev \
        uuid-dev \
        libffi-dev \
        tk-dev \
        wget \
        curl \
        git \
        make \
        sudo \
        bash-completion \
        tree \
        vim \
        software-properties-common && \
    mv /usr/bin/lsb_release /usr/bin/lsb_release.bak && \
    apt-get -y autoclean && \
    apt-get -y autoremove && \
    rm -rf /var/lib/apt-get/lists/*

ADD install_python.sh install_python.sh
RUN bash install_python.sh ${PYTHON_VERSION_TAG} && \
    rm -r install_python.sh Python-${PYTHON_VERSION_TAG}
# There is other stuff in the Dockerfile, such as setting up a non-root user, but not relevant here
CMD [ "/bin/bash" ]

and the install script lightly edited down to only include relevant parts ( install_python.sh )

#!/usr/bin/env bash

set -e

# This is being run as root and so sudo is not needed

CXX_VERSION="$(which gcc)"

function download_cpython () {
    # 1: the version tag
    printf "\n### Downloading CPython source as Python-${1}.tgz\n"
    wget "https://www.python.org/ftp/python/${1}/Python-${1}.tgz" &> /dev/null
    tar -xvzf "Python-${1}.tgz" > /dev/null
    rm "Python-${1}.tgz"
}

function build_cpython () {
    # 1: the prefix to be passed to configure
    #    c.f. https://docs.python.org/3/using/unix.html#python-related-paths-and-files
    # 2: the path to the version of gcc to be used
    # 3: the Python version being built

    # https://docs.python.org/3/using/unix.html#building-python
    # https://github.com/python/cpython/blob/3.7/README.rst
    printf "\n### ./configure\n"
    # --with-threads is removed in Python 3.7 (threading already on)
    ./configure --prefix="${1}" \
        --exec_prefix="${1}" \
        --with-cxx-main="${2}" \
        --enable-optimizations \
        --with-lto \
        --enable-loadable-sqlite-extensions \
        --enable-ipv6 \
        CXX="${2}"
    printf "\n### make -j4\n"
    make -j4
    printf "\n### make install\n"
    make install
}

function main() {
    # 1: the Python version tag

    PYTHON_VERSION_TAG=3.7.2

    if [[ $# -gt 0 ]]; then
        PYTHON_VERSION_TAG="${1}"
    fi

    download_cpython "${PYTHON_VERSION_TAG}"
    cd Python-"${PYTHON_VERSION_TAG}"
    build_cpython /usr "${CXX_VERSION}" "${PYTHON_VERSION_TAG}"
}

main "$@" || exit 1

Any thoughts or feedback on this issue would be very much appreciated.

To be slightly verbose, the output from docker build contains

0:21:41 load avg: 0.89 [335/416] test_sys_setprofile
0:21:42 load avg: 0.89 [336/416] test_sys_settrace
unhandled exception during asyncio.run() shutdown
task: <Task finished coro=<<async_generator_athrow without __name__>()> exception=RuntimeError("can't send non-None value to a just-started coroutine")>
RuntimeError: can't send non-None value to a just-started coroutine
unhandled exception during asyncio.run() shutdown
task: <Task finished coro=<<async_generator_athrow without __name__>()> exception=RuntimeError("can't send non-None value to a just-started coroutine")>
RuntimeError: can't send non-None value to a just-started coroutine
^[[0m0:21:43 load avg: 0.89 [337/416] test_sysconfig
0:21:44 load avg: 0.89 [338/416] test_syslog
0:21:45 load avg: 0.89 [339/416] test_tarfile
# ...
# and then later
# ...
0:28:10 load avg: 1.41 [415/416] test_zipimport_support
0:28:12 load avg: 1.46 [416/416] test_zlib

Total duration: 28 min 14 sec
Tests result: FAILURE

Edit: This is also happening with Python 3.7.3

Answer 1

While I still don't have a good answer as to why this was happening with builds of Python 3.7 I can report that this issues seems to be fixed in Python 3.8.1 . The Dockerfile and install_python.sh used (GitHub link for them here ) are basically the same as the ones in the question with the exception of the tag being changed to 3.8.1 .

With that the tests report succeeding when built in a ubuntu:bionic Docker image. The RuntimeError s from the question have been resolved

0:00:08 load avg: 6.49 [ 92/423] test_docxmlrpc passed
0:00:08 load avg: 6.49 [ 93/423] test_pickletools passed
0:00:08 load avg: 6.49 [ 94/423] test_contextlib_async passed
Task was destroyed but it is pending!
task: <Task pending name='Task-9' coro=<<async_generator_athrow without __name__>()>>
Task was destroyed but it is pending!
task: <Task pending name='Task-11' coro=<<async_generator_athrow without __name__>()>>
Task was destroyed but it is pending!
task: <Task pending name='Task-16' coro=<<async_generator_athrow without __name__>()>>
0:00:08 load avg: 6.49 [ 95/423] test_dynamic passed
0:00:08 load avg: 6.49 [ 96/423] test_userdict passed
0:00:08 load avg: 6.49 [ 97/423] test_sys_settrace passed
# ...
# and then later
# ...
0:02:42 load avg: 3.63 running: test_concurrent_futures (1 min 33 sec)
0:03:12 load avg: 3.09 running: test_concurrent_futures (2 min 3 sec)
0:03:26 load avg: 2.47 [423/423] test_concurrent_futures passed (2 min 17 sec)

== Tests result: SUCCESS ==

409 tests OK.

14 tests skipped:
    test_devpoll test_gdb test_ioctl test_kqueue test_msilib
    test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
    test_winconsoleio test_winreg test_winsound test_zipfile64

Total duration: 3 min 26 sec
Tests result: SUCCESS

As I can't see any reason to not upgrade from Python 3.7 to Python 3.8 (unless you're using a library that hasn't released a Python 3.8 wheel yet) I'm willing to say that just updating your Python runtime candidate is a good enough "fix" to mark this as resolved.