Unable to pip install in Docker image as agent through Jenkins declarative pipeline

Question

I have yet another issue with permissions running Docker through Jenkins declarative pipeline. I want to build and publish a Python package through a Jenkins job in a Docker container:

pipeline {

  agent {
    docker {
      image 'python:3.7'
      label 'docker && linux'
    }
  }

  environment {
    PACKAGE_VERSION = readFile 'VERSION'
  }

  stages {

    stage('Package') {
      steps {
        sh 'python -V'
        sh 'python -m pip install -r requirements.txt --user --no-cache'
        sh 'python setup.py sdist'
      }
    }

    stage('Deploy') {
      steps {
        ...
      }
    }

  }

  post {
    always {
      cleanWs()
    }
  }

}

However, I am not allowed to pip install due to a PermissionError :

+python -m pip install -r requirements.txt --user --no-cache Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (40.0.0) Collecting pytest (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/9e/a1/8166a56ce9d89fdd9efcae5601e71758029d90e5644e0b7b6eda07e67c35/pytest-3.7.0-py2.py3-none-any.whl (202kB) Collecting py>=1.5.0 (from pytest->-r requirements.txt (line 2)) Downloading https://files.pythonhosted.org/packages/f3/bd/83369ff2dee18f22f27d16b78dd651e8939825af5f8b0b83c38729069962/py-1.5.4-py2.py3-none-any.whl (83kB) Collecting more-itertools>=4.0.0 (from pytest->-r requirements.txt (line 2)) Downloading https://files.pythonhosted.org/packages/79/b1/eace304ef66bd7d3d8b2f78cc374b73ca03bc53664d78151e9df3b3996cc/more_itertools-4.3.0-py3-none-any.whl (48kB) Collecting pluggy>=0.7 (from pytest->-r requirements.txt (line 2)) Downloading https://files.pythonhosted.org/packages/f5/f1/5a93c118663896d83f7bcbfb7f657ce1d0c0d617e6b4a443a53abcc658ca/pluggy-0.7.1-py2.py3-none-any.whl Collecting six>=1.10.0 (from pytest->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl Collecting atomicwrites>=1.0 (from pytest->-r requirements.txt (line 2)) Downloading https://files.pythonhosted.org/packages/0a/e8/cd6375e7a59664eeea9e1c77a766eeac0fc3083bb958c2b41ec46b95f29c/atomicwrites-1.1.5-py2.py3-none-any.whl Collecting attrs>=17.4.0 (from pytest->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/41/59/cedf87e91ed541be7957c501a92102f9cc6363c623a7666d69d51c78ac5b/attrs-18.1.0-py2.py3-none-any.whl Installing collected packages: py, six, more-itertools, pluggy, atomicwrites, attrs, pytest

Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/.local' Check the permissions.

How do I fix these permissions?

Answer 1

I have found what I myself would think is the prettier solution:

stage("Python Test") {
  agent { 
    docker {
      label "docker && linux" 
      image "python:3.7"
    }
  }
  steps {
    withEnv(["HOME=${env.WORKSPACE}"]) {
      sh "pip install -r requirements.txt --user"
      # python stuff
    }
  }
  post {
    cleanup {
      cleanWs()
    }
  }
}

This workaround steers completely around the issue itself, installing the packages at user level. The issue here was that the HOME-directory was not initially writeable either, thus overwriting the HOME directory.

Answer 2

I had a very similar pipeline that I was running right after setting up Docker agents on my Jenkins system, so I thought my setup was wrong. Using the comments in your thread, I cooked up this solution:

First, you'll need to be root inside your container, so change your agent declaration to be similar to this:

agent {
    docker {
        image "python:3.7"
        args '--user 0:0'
    }
}

Now I was able to use pip install ! However, subsequent runs of the job would try to run git clean and fail since the built files inside the container were created by root. To fix that, I ran the clean command inside the container as my last step:

steps {
    sh 'git clean -fdx'
}

Update:

I found a problem where a failed build wouldn't clean up and killed all of the builds after it. To fix this, I put the clean action as a post-build task that always runs:

post {
    cleanup {
        cleanWs()
    }
}

Answer 3

withEnv(["HOME=${env.WORKSPACE}"]) {} is the key. Also, it can be used like:

node {
    def customImage = docker.build(...)
    customImage.inside {
        withEnv(["HOME=${env.WORKSPACE}"]) {
            sh '...'
        }
    }
}

Answer 4

You can try executing it as sudo:

 stage('Package') {
      steps {
        sh '''
            python -V
            sudo python -m pip install -r requirements.txt --user --no-cache
            sudo python setup.py sdist
           '''
      }
    }

You may have issues due Jenkins is not able to run command as sudo in that case I will recommend you to follow the steps mentioned in this article or in this SO question