Gitlab CI - server gets 'killed' before Cypress tests can run

Question

. 。 Answers to this question are eligible for a +500 reputation bounty. red house 87 wants to to this question. 这个问题。

I am running a CI pipeline in Gitlab which runs some Cypress integration tests as part of the testing stage. The tests are working absolutely fine on my machine locally but when I try and run them in Gitlab CI it appears that the Gitlab runner is killing my local server before I can run my Cypress tests against it. Here is my Gitlab config:

variables:
  API_BASE_URL: https://t.local.um.io/api
  CYPRESS_API_BASE_URL: https://t.local.um.io/api
  npm_config_cache: '$CI_PROJECT_DIR/.npm'
  CYPRESS_CACHE_FOLDER: '$CI_PROJECT_DIR/cache/Cypress'

cache:
  paths:
    - node_modules/
    - cache/Cypress

stages:
  - install
  - build
  - tests

install:
  image: cypress/browsers:node14.15.0-chrome86-ff82
  stage: install
  cache:
    key: 'e2eDeps'
    paths:
      - node_modules/
      - cache/Cypress/
  script:
    - npm ci

build:
  stage: build
  dependencies:
    - install
  script:
    - npm run build
  artifacts:
    expire_in: 1 days
    when: on_success

tests:
  image: cypress/browsers:node14.15.0-chrome86-ff82
  stage: tests
  script:
    - npm ci
    - npm run test:ci

And here are the relevant package.json scripts that the above config runs in CI:

  "scripts": {
    "build": "webpack --config webpack.prod.js",
    "dev": "webpack serve --config webpack.dev.js",
    "start:ci": "export NODE_OPTIONS=--max_old_space_size=4096 serve dist --no-clipboard --listen ${PORT:-3000}",
    "test": "cross-env NODE_ENV=test && npm run test:cypress && npm run test:jest",
    "test:ci": "cross-env NODE_ENV=test && start-server-and-test start:ci http-get://localhost:3000 test",
    "test:cypress": "cypress run --headless --browser chrome",
    "test:jest": "jest",
  },

It is the final stage tests that is currently failing. Here is the console output from the Gitlab runner, you can see where it says 'killed' then 'err no 137' it appears that it just stops the start:ci process which is what runs my local server so the integration tests can run against them.

Finally here is a small snippet of my test, I use the cy.visit command which never responds as the server is killed:

describe('Code entry page - API responses are managed correctly', () => {
  beforeEach(() => {
    cy.visit(routes.APP.HOME); // this just times out
  });
...

EDIT I have tried running the test:ci script inside of the exact same docker container that it uses ( cypress/browsers:node14.15.0-chrome86-ff82 ) locally (not in gitlabci) and it works no problem. The issue must lay with Gitlab surely?

Answer 1

Error 137 typically means that your Docker container was killed due to not having sufficient resources. As I mentioned in the comments, your current container is running with 4GB of memory. Since you are defining no tag keys within your CI/CD, you are likely running on GitLab's Linux runner cloud , which runs using a n1-standard-1 instance on GCP, which is limited to 3.75 GB of ram. Essentially, as soon as your test container starts, it instantly consumes all the memory available on the runner and your container is killed.

To get around the memory limitation, you must run your own gitlab-runner. There is no way to run using a higher amount of memory on the shared runner cloud. You can test this fairly easily by spinning up a gitlab-runner on your local machine ( see instructions here for installing a gitlab runner). Once you've installed your runner, register your runner to the high-memory tag, then update your CI/CD to use that tag using the following syntax on that last job:

tests:
  image: cypress/browsers:node14.15.0-chrome86-ff82
  stage: tests
  tags:
    - high-memory
  script:
    - npm ci
    - npm run test:ci

Your jobs are allowed to use as much memory as your runner has allocated. If your machine has 8Gb of memory, the jobs will be allowed to use up to 8Gb of memory.

If your machine doesn't have enough memory by itself, you could always temporarily spin up a cloud instance with sufficient memory. You could try a digital ocean droplet with 16GB of memory for 0.11c / hour, for example. This would allow you to run an instance for a couple of hours to test out the solution before you determine what's viable long-term.