No internet connectivity inside docker container running inside kubernetes with weave as networking

Question

I have a kubernetes cluster that is running on AWS EC2 instances and weave as networking(cni). I have disabled the docker networking(ipmask and iptables) as it is managed by weave(to avoid network conflicts).

I have deployed my Jenkins on this cluster as K8s pod and this jenkins uses jenkins kubernetes plugin to spawn dynamic slaves based on pod and container template which I have defined. These slaves container have docker client in it which connects to the host docker engine via docker.sock

So when I run any job in Jenkins it starts a slave and on this it clones a git repo and starts building the Dockerfile present inside the repo.

My sample dockerfile looks like this:

FROM abc:123
RUN yum update

So when container starts building this it tries connecting to redhat repo to update the local repo and fails here. To debug I logged in to this container and try wget/CURL some packages and finds that there is no internet connectivity in this container.

I suspect that while building docker starts intermediate containers and those containers are not managed by weave so they do not have internet connectivity.

Need suggestions.

Related question: Internet connection inside Docker container in Kubernetes

Answer 1

Ok finally after lot of struggle I find the solution.
So when ever K8s starts a pod it starts a sidecart container whose role is basically to provide network to pod containers.
So while running docker build if I pass it's container ID as network then my intermediate contexts start getting internet connectivity via this container. So changes looks something like this:

docker build -t "some name" --network container:\$(docker ps | grep \$(hostname) | grep k8s_POD | cut -d\" \" -f1) -f infra/docker/Dockerfile .

Hope this helps. :D

Answer 2

You can try to attach weave networking dynamically as a part of your build job. Is it definitely possible to change active network of container on the flight with weave.

Maybe you will need to use some additional container with Weave Docker Api Proxy or you can use a different way to communicate with Weave network on your nodes.

So, the main idea is just attach your containers where you running builds to the Kubernetes pods network, where you have an external access.

Also, and maybe it will be better, you can create another one Weave virtual network with access to the Internet and attach your contenders to it.

Answer 3

You're right - the docker build process runs in a different context, and Weave Net doesn't attach those automatically.

Even more complicated, Kubernetes will connect via CNI whereas Docker has its own plugin API. I believe it's possible to have both on a machine at the same time, but rather complicated.

Maybe look at some of the ways to build images without using Docker ?