Istio Pods Not Coming Up

Question

I have Installed Istio-1.8.3 via Rancher UI long back and Istio Pods and

Ingress Gateway Pods are Up and Running and My Application is getting served by Istio.

Now, Recently We have Upgraded the K8's Cluster version from 1.21 to 1.22 and then 1.22 to 1.23.

Once We restart the kubelet, Istio Pods used to come up with No Issues.

Now, Because of few Issues We have rebooted the Node and Istio Pods got restarted, They are in Running State but Readiness Probe is getting failed.

The Error I was able to find is

failed to list CRDs: the server could not find the requested resource

Below are the full logs of Istio Pod.

 stream logs failed container "discovery" in pod "istiod-5fbc9568cd-qgqkk" is waiting to start: ContainerCreating for istio-system/istiod-5fbc9568cd-qgqkk (discovery) 2022-06-27T05:35:32.772949Z info FLAG: --log_rotate_max_age="30" 2022-06-27T05:35:32.772952Z info FLAG: --log_rotate_max_backups="1000" 2022-06-27T05:35:32.772955Z info FLAG: --log_rotate_max_size="104857600" 2022-06-27T05:35:32.772958Z info FLAG: --log_stacktrace_level="default:none" 2022-06-27T05:35:32.772963Z info FLAG: --log_target="[stdout]" 2022-06-27T05:35:32.772971Z info FLAG: --mcpInitialConnWindowSize="1048576" 2022-06-27T05:35:32.772974Z info FLAG: --mcpInitialWindowSize="1048576" 2022-06-27T05:35:32.772977Z info FLAG: --mcpMaxMsgSize="4194304" 2022-06-27T05:35:32.772980Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2022-06-27T05:35:32.772982Z info FLAG: --monitoringAddr=":15014" 2022-06-27T05:35:32.772985Z info FLAG: --namespace="istio-system" 2022-06-27T05:35:32.772988Z info FLAG: --networksConfig="/etc/istio/config/meshNetworks" 2022-06-27T05:35:32.772999Z info FLAG: --plugins="[authn,authz,health]" 2022-06-27T05:35:32.773002Z info FLAG: --profile="true" 2022-06-27T05:35:32.773005Z info FLAG: --registries="[Kubernetes]" 2022-06-27T05:35:32.773008Z info FLAG: --resync="1m0s" 2022-06-27T05:35:32.773011Z info FLAG: --secureGRPCAddr=":15012" 2022-06-27T05:35:32.773013Z info FLAG: --tlsCertFile="" 2022-06-27T05:35:32.773016Z info FLAG: --tlsKeyFile="" 2022-06-27T05:35:32.773018Z info FLAG: --trust-domain="" 2022-06-27T05:35:32.801976Z info klog Config not found: /var/run/secrets/remote/config[] 2022-06-27T05:35:32.803516Z info initializing mesh configuration ./etc/istio/config/mesh 2022-06-27T05:35:32.804499Z info mesh configuration: { "proxyListenPort": 15001, "connectTimeout": "10s", "protocolDetectionTimeout": "0s", "ingressClass": "istio", "ingressService": "istio-ingressgateway", "ingressControllerMode": "STRICT", "enableTracing": true, "defaultConfig": { "configPath": "./etc/istio/proxy", "binaryPath": "/usr/local/bin/envoy", "serviceCluster": "istio-proxy", "drainDuration": "45s", "parentShutdownDuration": "60s", "discoveryAddress": "istiod.istio-system.svc:15012", "proxyAdminPort": 15000, "controlPlaneAuthPolicy": "MUTUAL_TLS", "statNameLength": 189, "concurrency": 2, "tracing": { "zipkin": { "address": "zipkin.istio-system:9411" } }, "envoyAccessLogService": { }, "envoyMetricsService": { }, "proxyMetadata": { "DNS_AGENT": "" }, "statusPort": 15020, "terminationDrainDuration": "5s" }, "outboundTrafficPolicy": { "mode": "ALLOW_ANY" }, "enableAutoMtls": true, "trustDomain": "cluster.local", "trustDomainAliases": [ ], "defaultServiceExportTo": [ "*" ], "defaultVirtualServiceExportTo": [ "*" ], "defaultDestinationRuleExportTo": [ "*" ], "rootNamespace": "istio-system", "localityLbSetting": { "enabled": true }, "dnsRefreshRate": "5s", "certificates": [ ], "thriftConfig": { }, "serviceSettings": [ ], "enablePrometheusMerge": true } 2022-06-27T05:35:32.804516Z info version: 1.8.3-e282a1f927086cc046b967f0171840e238a9aa8c-Clean 2022-06-27T05:35:32.804699Z info flags: 2022-06-27T05:35:32.804706Z info initializing mesh networks 2022-06-27T05:35:32.804877Z info mesh networks configuration: { "networks": { } } 2022-06-27T05:35:32.804938Z info initializing mesh handlers 2022-06-27T05:35:32.804949Z info initializing controllers 2022-06-27T05:35:32.804952Z info No certificates specified, skipping K8S DNS certificate controller 2022-06-27T05:35:32.814002Z error kube failed to list CRDs: the server could not find the requested resource 2022-06-27T05:35:33.816596Z error kube failed to list CRDs: the server could not find the requested resource 2022-06-27T05:35:35.819157Z error kube failed to list CRDs: the server could not find the requested resource 2022-06-27T05:35:39.821510Z error kube failed to list CRDs: the server could not find the requested resource 2022-06-27T05:35:47.823675Z error kube failed to list CRDs: the server could not find the requested resource 2022-06-27T05:36:03.827023Z error kube failed to list CRDs: the server could not find the requested resource 2022-06-27T05:36:35.829441Z error kube failed to list CRDs: the server could not find the requested resource 2022-06-27T05:37:35.831758Z error kube failed to list CRDs: the server could not find the requested resource

Answer 1

Upgrading Istio Pilot and Istio Ingress Gateway from 1.8.3 to 1.10.2 will work.

https://github.com/istio/istio/issues/34665

Answer 2

Istio version 1.8.x is too old version for kubernetes 1.23. You can refer istio documentation for k8s and istio combinations support and upgrade istio