kubernetes 上的 Jenkins 动态从站 - 构建时间非常长

Question

Background - Migrating a project from CircleCI to Jenkins.

Project technology - typescript (node.js)

I have deployed a Jenkins on a newly baked GKE cluster using the Jenkins official helm chart and leveraging the benefits of dynamic slaves.

The issue I am facing is with one of my application, it is a group of 4 microservice which build and deployed together as a project.

Since all apps build and ship together I have set up a Jenkins parallel build pipeline that pulls the repo and builds all the applications in parallel to save the build time(copied the same logic from the existing CircleCI setup).

In CircleCI it normally takes five to seven minutes to build the app whereas in Jenkins it is taking more than 20 minutes.

I doubt I have the limitation of the resources on the node and increased to a very high spec node and then monitored using the kubectl top pods command and I notice it never reaches more than 3 CPU during the entire build process.

For further debugging, I thought it could be the IOPS issues as the project is pulling a lot of node modules and I have changed the node disk to SSD for testing but no luck.

For further debugging, I started provisioning a dynamic PV with every slave that Jenkins Sprawns and no luck again.

I am not sure what I am missing I checked the docker stats, Kubernetes logs but everything looks normal.

I am ruining docker build like this(4 different applications):

docker build --build-arg NODE_HEAP_SIZE=8096 --build-arg NPM_TOKEN=$NPM_TOKEN -f "test/Dockerfile" -t "test:123"

This is how my Dockerfile looks like:

FROM node:10.19.0 AS node
WORKDIR /etc/xyz/test

COPY --from=gcr.io/berglas/berglas:0.5.0 /bin/berglas /usr/local/bin/berglas
COPY docker-entrypoint.sh /

ENTRYPOINT ["/docker-entrypoint.sh"]

#
# development stage used in conjunction with docker-compose for local development
#

FROM node AS dev
ENV NODE_ENV="development"


COPY new/package.json new/package-lock.json ../new/
RUN (cd ../new && npm install)

COPY brand/package.json brand/package-lock.json ../brand/
RUN (cd ../brand && npm install)

COPY chain/package.json chain/package-lock.json ./
RUN npm install

COPY chain ./
COPY new ../new/
COPY brand ../brand/

#
# production stage that compiles and runs production artifacts
#

FROM dev AS prod
ENV NODE_ENV="production"

ARG NODE_HEAP_SIZE="4096"
RUN NODE_OPTIONS="--max-old-space-size=${NODE_HEAP_SIZE}" npm run build:prod

To verify the network bandwidth on the nodes I have started a ubuntu container and did the network test and it is up to the mark.

I even tried passing the -cache-from to improve the caching during the build but no luck here as well.

I have even tried changing the NODE_HEAP_SIZE to a very high value but did not get any improvements.

I have seen the maximum time is going in npm install or npm ci or npm run build

Adding further investigation:

I have tried building the same steps on VM and also by spinning up a docker container on the same VM and tried to run the docker build inside, it is taking significantly less time than running in Jenkins dynamic slaves. The time difference is more or less double on dynamic slaves.

The maximum time is going in npm install and npm ci steps.

I don't know understand how CircleCi is able to build it faster.

Can someone help me with what else should I debug?

Answer 1

Without checking logs it is hard to say what is happening in Jenkins. Please take a look this article about global jenkins logs and configuring additional log recorders.

I had similar problem with dynamic jenkins slaves in AWS, because "Amazon EC2" plugin's developers changed security settings and it took ~15 minutes for checking ssh keys.