Running tests as usual against docker containers or dockerize tests?

Question

I'm new to Docker and was reading up on Docker. It's a great way to test systems in a self contained and reproducible standardized configuration (when done correctly).

However, in all the things I've read, there doesn't seem to be too much emphasis on how the testing should occur with docker containers. The docker is used to "contain" the infrastructure and application (code) for easy testing (as well as deployment). But sometimes test codebases are be large and not so simple too. And one can have a test codebase for API tests, another for UI, etc.

What is or should be (as determined at some point) the standard practice for testing docker containers/deployments of your applications/infrastructure? Should:

• the test code be deployed the old conventional way, as file repository you pull from somewhere and just then run on Jenkins server/slave or one's localhost for dev/QA testing/debugging, with the tests targeting apps in docker container?
• dockerize the whole test code base as a self contained container and then using that container to launch/execute tests against the other containers that have the app code/system infrastructure?
• combine the tests as part of the individual docker containers themselves to be run when/as needed. But I would think this works best only for unit tests that really pair with the container that holds the matching app code. Integration, UI, system level tests are different in association to app modules within the system.

The only reason I can think of that make dockerizing tests perhaps useful is that it's a single container with all the tests you need and the matching test infrastructure (all the test platform/language dependencies) so that one can deploy and run tests anywhere together with the matching app code containers. Saves one from having to set up test infrastructure when/as needed. But no seems to have blogged about such a thing for dockerized tests.

Answer 1

I prefer your option (3) ie to include test code in the production deployable artifact (the docker image)

Will quote Alister Scott from GTAC 2015 which I attended:

Don't be afraid to add testability specific features to your app that don't serve a functional purpose. I recently had to get new tyres on my car and realized that a lot of tyres have testability features called tread indicators. These don't serve a functional purpose

For integration and e2e tests, ie tests that require more than 1 docker image to be used, I prefer CI tool that, through docker-compose , and a separate git repo for these tests, orchestrates the creation of all containers that are needed for the larger test. Again the docker images used should be the exact same as for production except what varies is the configuration (eg environment variables) that make the tests point to test data and/or staging services.

Answer 2

I don't think you'd dockerize the tests themselves as the process that runs the tests.

For example if you need to run unit tests in php with phpunit, you would dockerize phpunit and use that to run tests against your code, and similarly for any other test framework you use (ie testng, junit... etc.)

Here's an example of the Dockerfile I use to encapsulate Java/Maven/TestNG for a Java test project which includes some selenium tests:

FROM maven:3-jdk-8

# Selectively add stuff we need
COPY pom.xml /usr/src/testng/

# Get a clean build immediately after and 'go-offline' to improve subsequent builds
RUN cd /usr/src/testng && mvn dependency:go-offline
COPY src /usr/src/testng/src
WORKDIR /usr/src/testng/

# Additional support files that's needed but not for the build
COPY supportfiles /usr/src/testng/supportfiles

CMD [ "mvn test" ]

Answer 3

Look closely at the screenshot I provide on this page: https://github.com/djangofan/karate-test-prime-example This is an example project I made that shows how to do a test container in context with a service and returning an exit code at the end of the tests.

Then when your code is released by a pipeline, before it goes live, your Docker container will include an exit code that allows your pipeline to decide whether to roll it back or not.