Tests fail ran by gitlab-ci, but not ran in bash

Question

I'm using gitlab-ci to automatically build a C++ project and run unit-tests written in python (it runs the daemon, and then communicates via the network/socket based interface).

The problem I'm finding is that when the tests are run by the GitLab-CI runner, they fail for various reasons (with one test, it stalls indefinitely on a particular network operation, on the other it doesn't receive a packet that should have been sent).

BUT: When I open up SSH and run the tests manually, they all work successfully (the tests also succeed on all of our developers' machines [linux/windows/OSX]).

At this point I've been trying to replicate enough of the build/test conditions that gitlab-ci is using but I don't really know any exact details, and none of my experiments have reproduced the problem.

I'd really appreciate help with either of the following:

• Guidance on running the tests manually outside of gitlab-ci, but replicating its environment so I can get the same errors/failures and debug the daemon and/or tests, OR
• Insight into why the test would fail when ran by GitLab-CI-Runner

Sidetrack 1: For some reason, not all the (mostly debugging) output that would normally be sent to the shell shows up in the gitlab-ci output.

Sidetrack 2: I also played around setting it up with jenkins, but one of the tests fails to even connect to the daemon, while the rest do it fine.

Answer 1

-i usually replicate the problem by using a docker container only for the runner and running the tests inside it, dont know if you have it setup like this =(. -Normally the test doesnt actually fail if you log in the container you will see he actually does everything but doesnt report back to the Gilab CI, dont freak out it does it job it simply does not say it.

PS: you can see if its actually running by checking the processes on the machine.

example: im running a gitlab ci with java and docker:

gitlab ci starts doing its thing then hangs at a download,meanwhile i log in the container and check that he is actually working and manages to upload my compiled docker image.