在 Gitlab CI 中运行基于硒的 pytest

Question

I'm trying to setup a project which should run e2e selenium based tests written in python inside a pipeline running on Gitlab CI. The goal is to use pytest-docker in order to use a docker-compose file to launch the needed applications before we can run the tests (This is just to justify why I'm using dind service and docker/compose image). However, I'm having issues with just running a simple test (which opens http://www.python.org and checks the title) inside Gitlab CI (locally runs fine).

So the test I'm trying to run is this:

import pytest
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from webdriver_manager.firefox import GeckoDriverManager
from pytest_bdd import scenarios, given, when, then, parsers


scenarios('../features/example.feature')


@pytest.fixture
def browser():
    s = Service(GeckoDriverManager().install())
    options = webdriver.FirefoxOptions()
    options.log.level = "TRACE"
    options.add_argument('--no-sandbox')
    options.add_argument('--headless')
    options.add_argument('--disable-gpu')
    b = webdriver.Firefox(service=s, options=options)
    b.implicitly_wait(10)
    yield b
    b.quit()


@when('the home page is displayed')
def home_displayed(browser):
    browser.get('http://www.python.org')


@then(parsers.parse('the page displays the title "{phrase}"'))
def page_displays_title(browser, phrase):
    assert "Python" in browser.title

And my gitlab-ci.yml looks like this:

acceptance-tests:
  stage: Acceptance Test
  image: docker/compose
  services:
    - docker:dind
  before_script:
    - echo 'https://dl-cdn.alpinelinux.org/alpine/v3.14/community' > /etc/apk/repositories
    - echo 'https://dl-cdn.alpinelinux.org/alpine/v3.14/main/' >> /etc/apk/repositories
    - apk update && apk add py-pip python3-dev libffi-dev openssl-dev gcc libc-dev rust cargo make firefox-esr
    - /usr/bin/python3.9 -m pip install --upgrade pip
    - pip install --no-cache-dir pipenv
    - pipenv install
  script:
    - pipenv run pytest src/backend/svfx22/tests/e2e/step_defs

Running this pipeline step in Gitlab CI results in the following stack trace:

$ pipenv run pytest src/backend/svfx22/tests/e2e/step_defs
============================= test session starts ==============================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
django: settings: svfx22.settings (from ini)
rootdir: /builds/msex20/svfx22/src/backend/svfx22, configfile: pytest.ini
plugins: bdd-4.1.0, cov-3.0.0, docker-0.10.3, django-4.4.0
collected 1 item
src/backend/svfx22/tests/e2e/step_defs/test_example.py F                 [100%]
=================================== FAILURES ===================================
___________________________ test_open_svf_home_page ____________________________
request = <FixtureRequest for <Function test_open_svf_home_page>>
    @pytest.mark.usefixtures(*function_args)
    def scenario_wrapper(request):
>       _execute_scenario(feature, scenario, request)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/pytest_bdd/scenario.py:165: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/pytest_bdd/scenario.py:136: in _execute_scenario
    _execute_step_function(request, scenario, step, step_func)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/pytest_bdd/scenario.py:100: in _execute_step_function
    kwargs = {arg: request.getfixturevalue(arg) for arg in get_args(step_func)}
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/pytest_bdd/scenario.py:100: in <dictcomp>
    kwargs = {arg: request.getfixturevalue(arg) for arg in get_args(step_func)}
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/_pytest/fixtures.py:581: in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/_pytest/fixtures.py:601: in _get_active_fixturedef
    self._compute_fixture_value(fixturedef)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/_pytest/fixtures.py:687: in _compute_fixture_value
    fixturedef.execute(request=subrequest)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/_pytest/fixtures.py:1072: in execute
    result = hook.pytest_fixture_setup(fixturedef=self, request=request)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/_pytest/fixtures.py:1126: in pytest_fixture_setup
    result = call_fixture_func(fixturefunc, request, kwargs)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/_pytest/fixtures.py:925: in call_fixture_func
    fixture_result = next(generator)
src/backend/svfx22/tests/e2e/conftest.py:72: in browser
    b = webdriver.Firefox(service=s, options=options)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/selenium/webdriver/firefox/webdriver.py:180: in __init__
    RemoteWebDriver.__init__(
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py:266: in __init__
    self.start_session(capabilities, browser_profile)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py:357: in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py:418: in execute
    self.error_handler.check_response(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7ffa69ad2f70>
response = {'status': 500, 'value': '{"value":{"error":"unknown error","message":"Process unexpectedly closed with status signal","stacktrace":""}}'}
    def check_response(self, response: Dict[str, Any]) -> None:
        """
        Checks that a JSON response from the WebDriver does not have an error.
    
        :Args:
         - response - The JSON response from the WebDriver server as a dictionary
           object.
    
        :Raises: If the response contains an error message.
        """
        status = response.get('status', None)
        if not status or status == ErrorCode.SUCCESS:
            return
        value = None
        message = response.get("message", "")
        screen: str = response.get("screen", "")
        stacktrace = None
        if isinstance(status, int):
            value_json = response.get('value', None)
            if value_json and isinstance(value_json, str):
                import json
                try:
                    value = json.loads(value_json)
                    if len(value.keys()) == 1:
                        value = value['value']
                    status = value.get('error', None)
                    if not status:
                        status = value.get("status", ErrorCode.UNKNOWN_ERROR)
                        message = value.get("value") or value.get("message")
                        if not isinstance(message, str):
                            value = message
                            message = message.get('message')
                    else:
                        message = value.get('message', None)
                except ValueError:
                    pass
    
        exception_class: Type[WebDriverException]
        if status in ErrorCode.NO_SUCH_ELEMENT:
            exception_class = NoSuchElementException
        elif status in ErrorCode.NO_SUCH_FRAME:
            exception_class = NoSuchFrameException
        elif status in ErrorCode.NO_SUCH_WINDOW:
            exception_class = NoSuchWindowException
        elif status in ErrorCode.STALE_ELEMENT_REFERENCE:
            exception_class = StaleElementReferenceException
        elif status in ErrorCode.ELEMENT_NOT_VISIBLE:
            exception_class = ElementNotVisibleException
        elif status in ErrorCode.INVALID_ELEMENT_STATE:
            exception_class = InvalidElementStateException
        elif status in ErrorCode.INVALID_SELECTOR \
                or status in ErrorCode.INVALID_XPATH_SELECTOR \
                or status in ErrorCode.INVALID_XPATH_SELECTOR_RETURN_TYPER:
            exception_class = InvalidSelectorException
        elif status in ErrorCode.ELEMENT_IS_NOT_SELECTABLE:
            exception_class = ElementNotSelectableException
        elif status in ErrorCode.ELEMENT_NOT_INTERACTABLE:
            exception_class = ElementNotInteractableException
        elif status in ErrorCode.INVALID_COOKIE_DOMAIN:
            exception_class = InvalidCookieDomainException
        elif status in ErrorCode.UNABLE_TO_SET_COOKIE:
            exception_class = UnableToSetCookieException
        elif status in ErrorCode.TIMEOUT:
            exception_class = TimeoutException
        elif status in ErrorCode.SCRIPT_TIMEOUT:
            exception_class = TimeoutException
        elif status in ErrorCode.UNKNOWN_ERROR:
            exception_class = WebDriverException
        elif status in ErrorCode.UNEXPECTED_ALERT_OPEN:
            exception_class = UnexpectedAlertPresentException
        elif status in ErrorCode.NO_ALERT_OPEN:
            exception_class = NoAlertPresentException
        elif status in ErrorCode.IME_NOT_AVAILABLE:
            exception_class = ImeNotAvailableException
        elif status in ErrorCode.IME_ENGINE_ACTIVATION_FAILED:
            exception_class = ImeActivationFailedException
        elif status in ErrorCode.MOVE_TARGET_OUT_OF_BOUNDS:
            exception_class = MoveTargetOutOfBoundsException
        elif status in ErrorCode.JAVASCRIPT_ERROR:
            exception_class = JavascriptException
        elif status in ErrorCode.SESSION_NOT_CREATED:
            exception_class = SessionNotCreatedException
        elif status in ErrorCode.INVALID_ARGUMENT:
            exception_class = InvalidArgumentException
        elif status in ErrorCode.NO_SUCH_COOKIE:
            exception_class = NoSuchCookieException
        elif status in ErrorCode.UNABLE_TO_CAPTURE_SCREEN:
            exception_class = ScreenshotException
        elif status in ErrorCode.ELEMENT_CLICK_INTERCEPTED:
            exception_class = ElementClickInterceptedException
        elif status in ErrorCode.INSECURE_CERTIFICATE:
            exception_class = InsecureCertificateException
        elif status in ErrorCode.INVALID_COORDINATES:
            exception_class = InvalidCoordinatesException
        elif status in ErrorCode.INVALID_SESSION_ID:
            exception_class = InvalidSessionIdException
        elif status in ErrorCode.UNKNOWN_METHOD:
            exception_class = UnknownMethodException
        else:
            exception_class = WebDriverException
        if not value:
            value = response['value']
        if isinstance(value, str):
            raise exception_class(value)
        if message == "" and 'message' in value:
            message = value['message']
    
        screen = None  # type: ignore[assignment]
        if 'screen' in value:
            screen = value['screen']
    
        stacktrace = None
        st_value = value.get('stackTrace') or value.get('stacktrace')
        if st_value:
            if isinstance(st_value, str):
                stacktrace = st_value.split('\n')
            else:
                stacktrace = []
                try:
                    for frame in st_value:
                        line = self._value_or_default(frame, 'lineNumber', '')
                        file = self._value_or_default(frame, 'fileName', '<anonymous>')
                        if line:
                            file = "%s:%s" % (file, line)
                        meth = self._value_or_default(frame, 'methodName', '<anonymous>')
                        if 'className' in frame:
                            meth = "%s.%s" % (frame['className'], meth)
                        msg = "    at %s (%s)"
                        msg = msg % (meth, file)
                        stacktrace.append(msg)
                except TypeError:
                    pass
        if exception_class == UnexpectedAlertPresentException:
            alert_text = None
            if 'data' in value:
                alert_text = value['data'].get('text')
            elif 'alert' in value:
                alert_text = value['alert'].get('text')
            raise exception_class(message, screen, stacktrace, alert_text)  # type: ignore[call-arg]  # mypy is not smart enough here
>       raise exception_class(message, screen, stacktrace)
E       selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status signal
/root/.local/share/virtualenvs/backend-MMqkD7aq/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py:243: WebDriverException
----------------------------- Captured stderr call -----------------------------
====== WebDriver manager ======
Current firefox version is 78.15
Get LATEST geckodriver version for 78.15 firefox
There is no [linux64] geckodriver for browser  in cache
Getting latest mozilla release info for v0.30.0
Trying to download new driver from https://github.com/mozilla/geckodriver/releases/download/v0.30.0/geckodriver-v0.30.0-linux64.tar.gz
Driver has been saved in cache [/root/.wdm/drivers/geckodriver/linux64/v0.30.0]
------------------------------ Captured log call -------------------------------
INFO     WDM:logger.py:26 
INFO     WDM:logger.py:26 ====== WebDriver manager ======
INFO     WDM:logger.py:26 Current firefox version is 78.15
INFO     WDM:logger.py:26 Get LATEST geckodriver version for 78.15 firefox
INFO     WDM:logger.py:26 There is no [linux64] geckodriver for browser  in cache
INFO     WDM:logger.py:26 Getting latest mozilla release info for v0.30.0
INFO     WDM:logger.py:26 Trying to download new driver from https://github.com/mozilla/geckodriver/releases/download/v0.30.0/geckodriver-v0.30.0-linux64.tar.gz
INFO     WDM:logger.py:26 Driver has been saved in cache [/root/.wdm/drivers/geckodriver/linux64/v0.30.0]
=========================== short test summary info ============================
FAILED src/backend/svfx22/tests/e2e/step_defs/test_example.py::test_open_svf_home_page
============================== 1 failed in 4.74s ===============================
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1

Can anyone help me regarding this? Am I missing something in my gitlab-ci.yml file? I've tried adding the selenium/standalone-firefox service, but with the same result.

Answer 1

This is a known issue with running Firefox in Docker. It was fixed in Firefox 84 .

For versions before Firefox 84, you have to increase the docker container's /dev/shm size (which is 64MB by default) to a size usable by Firefox, like 1g or larger. However, as far as I know, this isn't configurable using GitLab's shared runners. Therefore, you would need to self-host a GitLab runner for this or use another CI/browser provider for these older firefox versions.

Answer 2

As it turns out, the answer to my problem was adding the selenium/standalone-firefox service. I just had to properly configure my browser/driver in the test. So by adding:

  services:
    - selenium/standalone-firefox

in my gitlab-ci.yml and:

import pytest
from selenium import webdriver
from selenium.webdriver import Remote

@pytest.fixture
def browser(http_service):
    b = Remote(
        command_executor='http://selenium__standalone-firefox:4444/wd/hub',
        options=webdriver.FirefoxOptions()
    )
    b.implicitly_wait(10)
    yield b
    b.quit()

The problem went away.

Thank you for the answers.