Jenkins unable to create directory '/var/lib/jenkins/.ssh' when ssh into EC2

Question

Problem: During my Jenkins process I am able to establish a connection with the EC2 instance I want to copy files to but I keep getting the following errors:

Could not create directory '/var/lib/jenkins/.ssh'

Failed to add the host to the list of known hosts (/var/lib/jenkins/.ssh/known_hosts).

and

Host key verification failed.

Background: My Jenkins job is triggered by a github webhook after I push code to 'master' branch. Jenkins reads the repo's Jenkinsfile and creates a Docker agent to build the app and then deploy the built files to an EC2 container. During the deploy phase I use Jenkin's sshagent to establish a connection and then use commands to delete the old files and then copy the new files to the EC2.

pipeline {
  agent {
    docker {
      image 'node:buster'
      args '-p 20001-20100:3000'
      args '-v /etc/passwd:/etc/passwd'
    }
  }
   environment {
    CI = 'true'
    HOME = '.'
    npm_config_cache = 'npm-cache'
  }
  stages {
    stage('Install') {
      ...install code... <<<<<<<[works, no issues]   
    stage('Build') {
      ...build code... <<<<<<<[works, no issues]   
    }
    stage('Deploy') {
      parallel {
        stage('Deploy frontend') {
        ...deploy frontend code to S3 bucket... <<<<<<<[works, no issues]   
        }

        stage('Deploy backend') {
          steps {
            dir('backend') {
               sshagent(['code_commit_key']) {
                 sh 'ssh -o StrictHostKeyChecking=no ec2-user@ecx-xx-xx-x-xx.compute-1.amazonaws.com "whoami"' <<<<<[this return ec2-user after list of errors]
                 sh 'ssh -o StrictHostKeyChecking=no ec2-user@ecx-xx-xx-x-xx.compute-1.amazonaws.com "sudo su -; pm2 delete -s order-form-nestjs; rm -rf ./dist"' <<<<<[this returns list of errors]
                 sh 'scp -r ./dist/* ec2-user@ecx-xx-xx-x-xx.compute-1.amazonaws.com:/home/ec2-user' [this returns list of errors]
                 sh 'ssh -o StrictHostKeyChecking=no  ec2-user@ecx-xx-xx-x-xx.compute-1.amazonaws.com "sudo su -; pm2 start dist/main.js --name=backend-app-nestjs"' <<<<<[this returns list of errors]
                 echo 'Ssh successful'
               }
            }
          }
        }
      }
    }
  }
}```

Answer 1

Can you ssh your EC2s from your jenkins server? Can you check

Answer 2

Found my answer here :

By default, when the user is not specified, docker lauches [ sic ] the container with the user defined in the dockerfile which if not specified is root.

I added args '-u root:root -v /var/lib/jenkins/workspace/myworkspace:/tmp/' + ' -v /var/lib/jenkins/.ssh:/root/.ssh' to my docker agent code, and viola, success!:

agent {
    docker {
      image 'node:buster'
      args '-p 20001-20100:3000'
      args '-v /etc/passwd:/etc/passwd -v /etc/group:/etc/group'
      args '-u root:root -v /var/lib/jenkins/workspace/myworkspace:/tmp/' + ' -v /var/lib/jenkins/.ssh:/root/.ssh'
    }
  }