How to ignore certain scripts while testing flask app using pytest in gitlab CI/CD pipeline?

Question

I have a flask-restx folder with the following structure

.
├── app
│   ├── extensions.py
│   ├── __init__.py
│   └── pv_dimensioning
│       ├── controller.py
│       ├── __init__.py
│       ├── models
│       │   ├── dto.py
│       │   ├── __init__.py
│       │   ├── input_output_model.py
│       │   └── vendor_models.py
│       ├── services
│       │   ├── calculator.py
│       │   ├── database.py
│       │   ├── db_crud.py
│       │   ├── db_input_output.py
│       │   ├── __init__.py
│       │   └── processor.py
│       └── utils
│           ├── decode_verify_jwt.py
│           ├── decorator.py
│           └── __init__.py
├── config.py
├── main.py
├── package.json
├── package-lock.json
├── Pipfile
├── Pipfile.lock
├── README.md
├── serverless.yml
└── tests
    └── test_processor.py

The app is connected to a database, that's why there are many scripts in the app that require VPN connection.

I am writing tests using pytest which I will then run on the gitlab CI/CD to get a proper coverage . I would like to avoid or omit scripts that can only run when VPN is connected and write tests only for scripts that don't require VPN. (I have tests for the scripts that require VPN, but I just don't want to run them in the CI/CD pipeline)

The only script that doesn't require VPN is the processor.py and the test for that is in test_processor.py .

The scripts that I would like to avoid are in the .coveragerc :

[run]
omit =
    */site-packages/*
    */distutils/*
    tests/*
    /usr/*
    app/__init__.py
    app/extensions.py
    app/pv_dimensioning/models/*
    app/pv_dimensioning/utils/*
    app/pv_dimensioning/controller.py
    app/pv_dimensioning/services/calculator.py
    app/pv_dimensioning/services/database.py
    app/pv_dimensioning/services/db_crud.py
    app/pv_dimensioning/services/db_input_output.py

[html]
directory = htmlcov

the coverage part of .gitlab-ci.yml

stages:
  - coverage

coverage:
  image: python:3.7
  stage: coverage
  artifacts:
    paths:
      - htmlcov/
  before_script:
    - apt-get -y update
    - apt-get install curl
    - pip install pipenv
    - pipenv install --dev
  script:
    - pipenv run python -m coverage run -m pytest
    - pipenv run python -m coverage report -m
    - pipenv run python -m coverage html
  after_script:
    - pipenv run bash <(curl -s https://codecov.io/bash)

When I run the test in the pipeline, I get the following error:

$ pipenv run python -m coverage run -m pytest
============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /builds/EC/tool/dt-service
plugins: cov-2.11.0
collected 0 items / 1 error
==================================== ERRORS ====================================
___________________ ERROR collecting tests/test_processor.py ___________________
/usr/local/lib/python3.7/urllib/request.py:1350: in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
/usr/local/lib/python3.7/http/client.py:1277: in request
    self._send_request(method, url, body, headers, encode_chunked)
/usr/local/lib/python3.7/http/client.py:1323: in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
/usr/local/lib/python3.7/http/client.py:1272: in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
/usr/local/lib/python3.7/http/client.py:1032: in _send_output
    self.send(msg)
/usr/local/lib/python3.7/http/client.py:972: in send
    self.connect()
/usr/local/lib/python3.7/http/client.py:1439: in connect
    super().connect()
/usr/local/lib/python3.7/http/client.py:944: in connect
    (self.host,self.port), self.timeout, self.source_address)
/usr/local/lib/python3.7/socket.py:707: in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
/usr/local/lib/python3.7/socket.py:752: in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
E   socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
tests/test_processor.py:2: in <module>
    from app.pv_dimensioning.services.processor import PreProcessings
app/pv_dimensioning/__init__.py:4: in <module>
    from .controller import admin_crud_ns as admin_crud_namespace
app/pv_dimensioning/controller.py:5: in <module>
    from .services.calculator import DimensionCalculator
app/pv_dimensioning/services/calculator.py:3: in <module>
    from .database import DatabaseService
app/pv_dimensioning/services/database.py:6: in <module>
    from ..utils.decode_verify_jwt import verifier
app/pv_dimensioning/utils/decode_verify_jwt.py:12: in <module>
    with urllib.request.urlopen(keys_url) as f:
/usr/local/lib/python3.7/urllib/request.py:222: in urlopen
    return opener.open(url, data, timeout)
/usr/local/lib/python3.7/urllib/request.py:525: in open
    response = self._open(req, data)
/usr/local/lib/python3.7/urllib/request.py:543: in _open
    '_open', req)
/usr/local/lib/python3.7/urllib/request.py:503: in _call_chain
    result = func(*args)
/usr/local/lib/python3.7/urllib/request.py:1393: in https_open
    context=self._context, check_hostname=self._check_hostname)
/usr/local/lib/python3.7/urllib/request.py:1352: in do_open
    raise URLError(err)
E   urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
=========================== short test summary info ============================
ERROR tests/test_processor.py - urllib.error.URLError: <urlopen error [Errno ...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 1.62s ===============================

In the trace, it can be seen that the scripts that I am trying to ignore aren't being ignored. What is the mistake I am doing?

Answer 1

As I understand it, coverage is about reporting how much of your codebase is tested, not which tests to run. What you're doing is excluding things from a report, not stopping the data for the report being created.

What you should do is skip tests if you know they're going to fail (due to external configuration). Fortunately pytest provides for this with the skipif decorator.

I would create a function in tests/conftest.py which skips tests if the VPN is active. Something like:

import socket
import pytest

def _requires_vpn():
    has_vpn = False
    try:
        socket.gethostbyname("<some address only accessible in the VPN>")
        has_vpn = True
    except socket.error:
        pass

    return pytest.mark.skipif(not has_vpn, reason="access to the vpn is required")

requires_vpn = _requires_vpn()  # this is like the minversion example on the link

Then you can ignore entire test files adding this at the top:

pytestmark = requires_vpn

And you can skip specific tests by decorating with this.

@requires_vpn
def test_this_will_be_skipped():
    pass