Jenkins cannot start a docker agent container in declarative pipeline with specific run arguments

Question

I have the next setup:

• one amazon EC2 instance;
• Jenkins installed on it;
• Docker instance installed along with Jenkins;
• Docker instance configured as cloud on Jenkins;
• docker agent template is also configured and can be used when the pipeline calls it by label on the agent section.

What I want to accomplish is to use a docker container as the main agent, in a Jenkins pipeline. Now this agent needs to run some build steps and remain running until some other build plan stops the running containers and cleans things up. This is the pipeline script:

pipeline {
agent {
    docker {
        image 'localhost:5000/build_deploy_agent:base'
        label 'build-deploy-agent'
        args  '--network bridge -p 9102:9102'
    }
} 
stages {
    stage('Example Build') {
        steps {
            sh 'ps aux'
        }
    }
}
}

What happens is: Jenkins starts the container but it cannot be used to run the script from the stage. I receive three errors like this one:

docker inspect -f . localhost:5000/build_deploy_agent:base
/home/jenkins/workspace/view name/Build plan name@tmp/durable-9a8a9027/script.sh: 1: 
/home/jenkins/workspace/view name/Build plan name@tmp/durable-9a8a9027/script.sh: docker: not found

I am definitely doing something wrong here, but I am clueless.

Update 1

• the base docker image used is the 'jenkins/agent' image from the docker hub.
• below I am adding an image with the docker agent setup:

Answer 1

TL;DR

The server on which jenkins is running needs the docker-ce-cli and the user running the jenkins server should be in the docker group (if not running as root ).

---

This example uses everything in docker orchestrated using docker-compose .

.
├── docker-compose.yaml
├── Dockerfile
├── etc
│   └── nginx
│       └── conf.d
│           └── default.conf
└── plugins.txt

The docker-compose spec runs the jenkins server proxied by nginx :

version: '3.7'
services:
  nginx:
    image: 'nginx:1.19'
    container_name: 'nginx'
    restart: 'always'
    depends_on:
    - 'jenkins'
    ports:
    - '80:80'
    volumes:
    - 'jenkins:/var/jenkins_home'
    - './etc/nginx/conf.d/default.conf:/etc/nginx/conf.d/default.conf'
  jenkins:
    build:
      context: '.'
    container_name: 'jenkins'
    privileged: true
    restart: 'always'
    expose:
    - '50000'
    - '8080'
    volumes:
    - 'jenkins:/var/jenkins_home'
    - '/var/run/docker.sock:/var/run/docker.sock'
volumes:
  jenkins:

Note : the jenkins home is shared between the nginx server and the jenkins service for the userContent .

Note : the jenkins service shares the docker socket from the host.

A slightly modified nginx.conf from the doc :

upstream jenkins {
  keepalive 32;
  server jenkins:8080 max_fails=3;
}
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}
server {
  listen *:80;
  listen [::]:80;
  server_name _;

  charset utf-8;
  ignore_invalid_headers off;

  error_page   500 502 503 504  /50x.html;
  location = /50x.html {
      root   /usr/share/nginx/html;
  }
  location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
    rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
  }
  location /userContent {
    root /var/jenkins_home/;
    if (!-f $request_filename){
      rewrite (.*) /$1 last;
      break;
    }
    sendfile on;
  }
  location / {
    sendfile off;
    proxy_pass         http://jenkins;
    proxy_redirect     default;
    proxy_http_version 1.1;

    proxy_set_header   Connection        $connection_upgrade;
    proxy_set_header   Upgrade           $http_upgrade;

    proxy_set_header   Host              $host;
    proxy_set_header   X-Real-IP         $remote_addr;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_max_temp_file_size 0;

    client_max_body_size       10m;
    client_body_buffer_size    128k;

    proxy_connect_timeout      90;
    proxy_send_timeout         90;
    proxy_read_timeout         90;
    proxy_buffering            off;
    proxy_request_buffering    off;
    proxy_set_header Connection "";
  }
}

The jenkins image is extended with docker and the jenkins user is added to the ping group (the group ID is 999 which is the docker group ID on the host). The Dockerfile :

FROM docker:20.10.5-dind as docker
FROM jenkins/jenkins:alpine
USER root
COPY --from=docker /usr/local/bin/docker /usr/local/bin/docker
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt
RUN adduser jenkins ping
USER jenkins

Note : the jenkins image is pre-baked with the default suggested plugins plus the Docker and Docker Pipeline plugins. The plugins.txt :

github:1.33.1
pipeline-model-api:1.8.4
scm-api:2.6.4
mailer:1.32.1
workflow-support:3.8
font-awesome-api:5.15.2-2
pipeline-milestone-step:1.3.2
git:4.6.0
plain-credentials:1.7
resource-disposer:0.15
jackson2-api:2.12.1
jquery3-api:3.5.1-3
gradle:1.36
credentials:2.3.15
docker-workflow:1.26
workflow-scm-step:2.12
display-url-api:2.3.4
bootstrap4-api:4.6.0-2
antisamy-markup-formatter:2.1
command-launcher:1.5
pipeline-stage-tags-metadata:1.8.4
snakeyaml-api:1.27.0
pipeline-stage-view:2.19
script-security:1.76
okhttp-api:3.14.9
pipeline-stage-step:2.5
workflow-step-api:2.23
timestamper:1.11.8
pipeline-github-lib:1.0
token-macro:2.13
pam-auth:1.6
workflow-cps-global-lib:2.18
ws-cleanup:0.39
pipeline-model-definition:1.8.4
workflow-aggregator:2.6
jsch:0.1.55.2
matrix-auth:2.6.5
ssh-credentials:1.18.1
ant:1.11
jjwt-api:0.11.2-9.c8b45b8bb173
momentjs:1.1.1
trilead-api:1.0.13
durable-task:1.35
workflow-job:2.40
git-server:1.9
ssh-slaves:1.31.5
plugin-util-api:2.0.0
git-client:3.6.0
lockable-resources:2.10
checks-api:1.5.0
pipeline-input-step:2.12
cloudbees-folder:6.15
pipeline-build-step:2.13
popper-api:1.16.1-2
pipeline-graph-analysis:1.10
matrix-project:1.18
workflow-api:2.41
github-branch-source:2.9.7
workflow-basic-steps:2.23
apache-httpcomponents-client-4-api:4.5.13-1.0
workflow-multibranch:2.22
workflow-cps:2.90
ldap:1.26
build-timeout:1.20
echarts-api:5.0.1-1
pipeline-model-extensions:1.8.4
structs:1.22
junit:1.48
docker-java-api:3.1.5.2
docker-plugin:1.2.2
workflow-durable-task-step:2.38
credentials-binding:1.24
jdk-tool:1.5
bouncycastle-api:2.20
docker-commons:1.17
github-api:1.123
authentication-tokens:1.4
email-ext:2.82
branch-api:2.6.2
pipeline-rest-api:2.19
ace-editor:1.1
handlebars:1.1.1

You can bring up the environment using docker-compose ie: docker-compose up -d . You can access the jenkins server on http://localhost once the server is started.

Basic jenkins pipeline:

pipeline {
    agent {
        docker { 
            image 'ubuntu:20.04' 
            
        }
    }
    stages {
        stage('Test') {
            steps {
                sh 'ps -axf'
            }
        }
    }
}

... and console output:

Note : not using any labels or templates (irrelevant to the problem). The docker run is done from the jenkins host.