Pip install not working with jenkins?

Question

That is my Jenkinsfile.

pipeline {
    agent none
    stages {
        stage('Build') {
            agent {
                docker {
                    image 'python:3-alpine'
                }
            }
            steps {
                sh 'pip install --user -r requirements.txt'
                sh 'python WebChecker.py'
            }
            post {
                always {
                    junit 'output.xml'
                }
            }
        }
    }
}

When I run it in Jenkins, I get the follow

[urltester] Running shell script

+ pip install --user -r requirements.txt

The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Collecting beautifulsoup4==4.6.0 (from -r requirements.txt (line 1))

  Downloading https://files.pythonhosted.org/packages/9e/d4/10f46e5cfac773e22707237bfcd51bbffeaf0a576b0a847ec7ab15bd7ace/beautifulsoup4-4.6.0-py3-none-any.whl (86kB)

Collecting requests==2.18.4 (from -r requirements.txt (line 2))

  Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl (88kB)

Collecting junit-xml==1.8 (from -r requirements.txt (line 3))

  Downloading https://files.pythonhosted.org/packages/a6/2a/f8d5aab80bb31fcc789d0f2b34b49f08bd6121cd8798d2e37f416df2b9f8/junit-xml-1.8.tar.gz

Collecting urllib3<1.23,>=1.21.1 (from requests==2.18.4->-r requirements.txt (line 2))

  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)

Collecting idna<2.7,>=2.5 (from requests==2.18.4->-r requirements.txt (line 2))

  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)

Collecting certifi>=2017.4.17 (from requests==2.18.4->-r requirements.txt (line 2))

  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)

Collecting chardet<3.1.0,>=3.0.2 (from requests==2.18.4->-r requirements.txt (line 2))

  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)

Collecting six (from junit-xml==1.8->-r requirements.txt (line 3))

  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl

Installing collected packages: beautifulsoup4, urllib3, idna, certifi, chardet, requests, six, junit-xml

Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/.local'

Check the permissions.



script returned exit code 1

So then I do sudo pip install ....

And I get the following error:

[urltester] Running shell script

+ sudo python -m pip install --user -r requirements.txt

/Users/me/.jenkins/workspace/urltester@tmp/durable-e36d9731/script.sh: line 1: sudo: not found

script returned exit code 127

I then removed the sudo and tried using a virtual env:

pipeline {
    agent none
    stages {
        stage('Build') {
            agent {
                docker {
                    image 'python:3-alpine'
                }
            }
            steps {
                sh 'virtualenv venv --distribute'
                sh 'source venv/bin/activate '
                sh 'pip install --user -r requirements.txt'
                sh 'python WebChecker.py'
            }
            post {
                always {
                    junit 'output.xml'
                }
            }
        }
    }
}

But then I got the same thing as when I tried sudo, but this time virtualenv was not found.

My end goal is to be able to run my python script. This python script will generate an xml file in its same directory. Jenkins should then read this xml file. I tried using Docker, but did not get far with that. So what should I do?

Answer 1

As tftd wrote, change HOME to writable directory like:

pipeline {
    agent none
    stages {
        stage('Build') {
            agent {
                docker {
                    image 'python:3-alpine'
                }
            }
            steps {
                withEnv(["HOME=${env.WORKSPACE}"]) {
                    sh 'pip install --user -r requirements.txt'
                    sh 'python WebChecker.py'
                }
            }
            post {
                always {
                    junit 'output.xml'
                }
            }
        }
    }
}

Answer 2

You need to add virtualenv to PATH variable.

If you installed it using pip install virtualenv , it will be in pythonX.X/Lib/site-packages/

Sudo should also be added to PATH variable.

---

The error in first code snippet is because you did not have permissions to write to '/.local' . Try running it as administrator

Answer 3

Try to add argument -u root:root like this:

withDockerContainer(image: 'python:3.6', args:'-u root:root'){
        sh """
            pip install --user -r requirements.txt
            python WebChecker.py
        """
    }

Answer 4

I'm running into this now. What is not shown in the OPs original post is the command Jenkins uses to run Docker. It turns out to be:

docker run -t -d -u XXX:YYY -w /var/lib/jenkins/workspace/

where XXX is the UID of the user running jenkins, and YYY is the group id. In the python container, this UID is not recognized, hence there is no home directory. When 'pip install --user' tries to run, since there is no home directory, it defaults '/' which is not writeable.

I think this is a fallacy of Jenkins (see https://issues.jenkins-ci.org/browse/JENKINS-47026 ). The other CICD services out there seem to handle this just fine.

Yoichi Nakayama answer worked for me, and I'm upvoting it.