Pip too slow in gitlab-ci

Question

Since yesterday my gitlab-ci is too slow to download pip packages like:

• more_itertools
• importlib_metadata
• amqp
• kombu
• jmespath
• idna

Example:

And in the end I get this error: ERROR: Job failed: execution took longer than 1h0m0s seconds

Does anyone have any ideas?

Answer 1

Installing packages like this (whether it's OS packages though apt-get install... or pip, or anything else) is generally against best practices for CI/CD jobs because every job that runs will have to do the same thing, costing a lot of time as you run more pipelines. The alternative is to search for an existing image that has everything you need (possible but not likely with more dependencies), split up your job into pieces that might be solved by an image with just one or two dependencies, or create a custom docker image to use in your jobs.

Since you didn't post the full job output or your .gitlab-ci.yml file, I'm just assuming you're using the Docker executor for your runners, but if you're using shell , another solution is to install all the dependencies you need on the host running gitlab-runner as the shell executor will have access to them.

I just answered another question about creating a custom docker image for Gitlab CI that you can reference here: "Unable to locate package git" when running GitLab CI/CD pipeline

Once you've moved your package dependencies to a docker image, your job will pull in the already-built image and skip to the meat of your job.