EKS Anywhere Cluster cert-manager io-timeout

Question

First time trying EKS Anywhere docker provider deployment as given in below link https://anywhere.eks.amazonaws.com/docs/getting-started/local-environment/

It gets stuck at 'waiting for cert-manager'. Working on CentOS 7.System is behind proxy.

Installing cert-manager Version="v1.5.3+66e1acc"
Using Override="cert-manager.yaml" Provider="cert-manager" Version="v1.5.3+66e1acc"
Waiting for cert-manager to be available...
Error: timed out waiting for the condition

Only cert-manager pods are not able to pull images

   NAMESPACE            NAME                                                              READY   STATUS             RESTARTS   AGE
  cert-manager         cert-manager-7988d4fb6c-bjhsv                                     0/1     ImagePullBackOff   0          5m54s
  cert-manager         cert-manager-cainjector-6bc8dcdb64-hvdx5                          0/1     ImagePullBackOff   0          5m55s
  cert-manager         cert-manager-webhook-68979bfb95-q8ttt                             0/1     ImagePullBackOff   0          5m54s
  kube-system          coredns-745c7986c7-2wrx5                                          1/1     Running            0          5m57s
  kube-system          coredns-745c7986c7-kx594                                          1/1     Running            0          5m57s
  kube-system          etcd-dev-cluster-eks-a-cluster-control-plane                      1/1     Running            0          5m52s
  kube-system          kindnet-4jcvt                                                     1/1     Running            0          5m57s
  kube-system          kube-apiserver-dev-cluster-eks-a-cluster-control-plane            1/1     Running            0          5m52s
  kube-system          kube-controller-manager-dev-cluster-eks-a-cluster-control-plane   1/1     Running            0          5m52s
  kube-system          kube-proxy-4dk2r                                                  1/1     Running            0          5m57s
  kube-system          kube-scheduler-dev-cluster-eks-a-cluster-control-plane            1/1     Running            0          5m52s
  local-path-storage   local-path-provisioner-666bfc797f-nkhqf                           1/1     Running            0          5m57s

same images are getting pulled using docker pull

 public.ecr.aws/eks-anywhere/jetstack/cert-manager-webhook      v1.5.3-eks-a-6                 194bcfda671e   3 months ago    46MB
 public.ecr.aws/eks-anywhere/jetstack/cert-manager-controller   v1.5.3-eks-a-6                 1e6749016508   3 months ago    61.3MB
 public.ecr.aws/eks-anywhere/jetstack/cert-manager-cainjector   v1.5.3-eks-a-6                 45723d794a88   3 months ago    42.4MB

kubectl describe gives below (i/o timeout) error as well as 'server misbehaving' error

 Failed to pull image "public.ecr.aws/eks-anywhere/jetstack/cert-manager-controller:v1.5.3-eks-a-6": rpc error: code = Unknown desc = failed to pull and unpack image "public.ecr.aws/eks-anywhere/jetstack/cert-manager-controller:v1.5.3-eks-a-6": failed to resolve reference "public.ecr.aws/eks-anywhere/jetstack/cert-manager-controller:v1.5.3-eks-a-6": failed to do request: Head "https://public.ecr.aws/v2/eks-anywhere/jetstack/cert-manager-controller/manifests/v1.5.3-eks-a-6": dial tcp: lookup public.ecr.aws on 172.19.0.1:53: read udp 172.19.0.2:38941->172.19.0.1:53: i/o timeout

Answer 1

It was a proxy related issue. Resolved by adding proxy config in containerd service of docker container of node and restarting containerd service.

docker exec -it <container name> bash

Inside container

cd /etc/systemd/system/
mkdir containerd.service.d
touch http-proxy.conf
cat <<EOF >/etc/systemd/system/containerd.service.d/http-proxy.conf    
[Service]    
Environment="HTTP_PROXY=http://proxy ip:proxy port"    
Environment="HTTPS_PROXY=http://proxy ip:proxy port"    
Environment="NO_PROXY=${NO_PROXY:-localhost},${LOCAL_NETWORK}"    
EOF
systemctl daemon-reload
systemctl restart containerd