AWS EKS - kubectl logs [pod name]

Question

I have an EKS cluster running on AWS in us-east-1. I have three worker nodes running on us-east-1a, 1b, 1c respectively. All the pods running in us-east-1a allow me to login and check logs but the pods in other zones keep throwing an error:

Error from server: no preferred addresses found; known addresses: []

Any help appreciated. Basically using this setup for sharding mongodb in various zones.

Answer 1

In all workers

1) Edit the file:

/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Add or update:

Environment="KUBELET_EXTRA_ARGS=--node-ip='private-ip-value'"

For eg: Environment="KUBELET_EXTRA_ARGS=--node-ip=172.31.55.201"

2) systemctl daemon-reload

3) systemctl restart kubelet

Answer 2

The error message you're encountering, "Error from server: no preferred addresses found; known addresses: []", typically occurs when the Kube.netes API server is unable to determine an appropriate address for the requested pod. This can happen if the pod's IP address is not correctly registered or if there are.networking issues within your cluster.

To troubleshoot and resolve this issue, you can follow these steps:

1. Check Node Status: Verify that the worker nodes in the availability zones us-east-1b and us-east-1c are in a healthy state. You can use the following command to check the node status:

Ensure that all nodes are in the "Ready" state and have the appropriate IP addresses assigned.

2. Verify Pod Scheduling: Check if the pods in the availability zones us-east-1b and us-east-1c are successfully scheduled and running. Use the following command to verify the pod status:

Make sure that the pods in those availability zones are running without any issues.

3. Check Networking Configuration: Ensure that the.networking configuration within your EKS cluster is correctly set up. Specifically, check the following:

   • Verify that the VPC, su.nets, and security groups associated with your EKS cluster are properly configured and allow communication between nodes and pods in different availability zones.
   • Ensure that the EKS cluster's.networking components, such as VPC CNI, are functioning correctly. You can review the EKS cluster's.networking configuration and troubleshoot any.networking issues following the AWS documentation.

4. Review Cluster Logs: Examine the logs of your EKS cluster, including the control plane logs, to identify any potential errors or.networking-related issues. You can access the logs using the AWS Management Console or AWS CLI.

   • For AWS CLI, you can use the following command to retrieve the control plane logs:

      aws eks describe-cluster --name <cluster-name> --region <region> --query "cluster.logging.clusterLogging[?enabled==\`true\`].types[]" --output text

   • Once you identify the enabled log types, you can use the appropriate AWS CLI command to retrieve the logs. For example, to get the control plane logs:

      aws eks describe-cluster --name <cluster-name> --region <region> --query "cluster.logging.clusterLogging[?enabled==\`true\`].types[]" --output text

5. Verify DNS Resolution: Ensure that the DNS resolution is working correctly within your EKS cluster. The pods should be able to resolve the service names and IP addresses of other pods. Check if DNS resolution is functioning as expected.

By following these steps and reviewing the logs and.networking configuration, you should be able to identify and resolve the issue causing the error message when accessing logs from pods in the availability zones us-east-1b and us-east-1c.

I also found this open-source project on GitHub- Awesome-CloudOps-Automation . They are creating an open-source framework for writing Runbooks using Jupyter Notebooks. They have actions like Copy logs to S3 buckets and many more , that can be used to automate some of the AWS EKS related tasks.