Kubenetes：是否可以在 Kubernetes 集群中通过单个请求命中多个 Pod

Question

I want to clear cache in all the pods in my Kubernetes namespace. I want to send one request to the end-point which will then send a HTTP call to all the pods in the namespace to clear cache. Currently, I can hit only one pod using Kubernetes and I do not have control over which pod would get hit.

Even though the load-balancer is set to RR, continuously hitting the pods(n number of times, where n is the total number of pods) doesn't help as some other requests can creep in.

The same issue was discussed here, but I couldn't find a solution for the implementation: https://github.com/kubernetes/kubernetes/issues/18755

I'm trying to implement the clearing cache part using Hazelcast, wherein I will store all the cache and Hazelcast automatically takes care of the cache update.

If there is an alternative approach for this problem, or a way to configure kubernetes to hit all end-points for some specific requests, sharing here would be a great help.

Answer 1

Provided you got kubectl in your pod and have access to the api-server, you can get all endpoint adressess and pass them to curl:

kubectl get endpoints <servicename> \
        -o jsonpath="{.subsets[*].addresses[*].ip}" | xargs curl

Alternative without kubectl in pod:

the recommended way to access the api server from a pod is by using kubectl proxy: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod this would of course add at least the same overhead. alternatively you could directly call the REST api, you'd have to provide the token manually.

APISERVER=$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")
TOKEN=$(kubectl describe secret $(kubectl get secrets \
     | grep ^default | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' | tr -d " ")

if you provide the APISERVER and TOKEN variables, you don't need kubectl in your pod, this way you only need curl to access the api server and "jq" to parse the json output:

curl $APISERVER/api/v1/namespaces/default/endpoints --silent \
     --header "Authorization: Bearer $TOKEN" --insecure \
     | jq -rM ".items[].subsets[].addresses[].ip" | xargs curl

UPDATE (final version)

APISERVER usually can be set to kubernetes.default.svc and the token should be available at /var/run/secrets/kubernetes.io/serviceaccount/token in the pod, so no need to provide anything manually:

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token); \
curl https://kubernetes.default.svc/api/v1/namespaces/default/endpoints --silent \
     --header "Authorization: Bearer $TOKEN" --insecure \
     | jq -rM ".items[].subsets[].addresses[].ip" | xargs curl

jq is available here: https://stedolan.github.io/jq/download/ (< 4 MiB, but worth it for easily parsing JSON)

Answer 2

I have had the similar situation. Here is how I resolved it (I'm using a namespace other than "default").

Setup access to cluster Using RBAC Authorization

Access to API is done by creating a ServiceAccount , assign it to the Pod and bind a Role to it.

1.Create a ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-serviceaccount
  namespace: my-namespace

2.Create a Role : in this section you need to provide the list of resources and the list of actions you'd like to have access to. Here is the example where you'd like to list the endpoints and also get the details of a specific endpoint.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-role
  namespace: my-namespace
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list"]

3.Bind the role to the service account

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-role-binding
  namespace: my-namespace
subjects:
- kind: ServiceAccount
  name: my-serviceaccount
roleRef:
  kind: Role
  name: my-role
  apiGroup: rbac.authorization.k8s.io

4.Assign the service account to the pods in your deployment (it should be under template.spec)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  namespace: my-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
          app: my-pod
  template:
    metadata:
      labels:
        app: my-pod
    spec:
      serviceAccountName: my-serviceaccount
      containers:
      - name: my-pod
        ...

Access Clusters Using the Kubernetes API

Having all the security aspects set, you will have enough privilege to access the API within your Pod. All the required information to communicate with API Server is mounted under /var/run/secrets/kubernetes.io/serviceaccount in your Pod. You can use the following shell script (probably add it to your COMMAND or ENTRYPOINT of the Docker image).

#!/bin/bash
# Point to the internal API server hostname
API_SERVER=https://kubernetes.default.svc

# Path to ServiceAccount token
SERVICE_ACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICE_ACCOUNT}/namespace)

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICE_ACCOUNT}/token)

# Reference the internal certificate authority (CA)
CA_CERT=${SERVICE_ACCOUNT}/ca.crt

From this point forward, it is just simple REST API call. You can read these environment variables in any language of your choice and access to API.

Here is an example of listing the endpoint for your use case

# List all the endpoints in the namespace that Pod is running
curl --cacert ${CA_CERT} --header "Authorization: Bearer ${TOKEN}" -X GET \
  "${API_SERVER}/api/v1/namespaces/${NAMESPACE}/endpoints"

# List all the endpoints in the namespace that Pod is running for a deployment
curl --cacert ${CA_CERT} --header "Authorization: Bearer ${TOKEN}" -X GET \
  "${API_SERVER}/api/v1/namespaces/${NAMESPACE}/endpoints/my-deployment"

For more information on available API endpoints and how to call them, refer to API Reference .

Answer 3

For those of you trying to find an alternative, I have used hazelcast as distributed event listener. Added a similar POC on github: https://github.com/vinrar/HazelcastAsEventListener

Answer 4

I fixed this problem by usingthis script . You just have to write the equivalent command to make the API call. I used curl to do that.

Following is the usage of the script:

function usage {
    echo "usage: $PROGNAME [-n NAMESPACE] [-m MAX-PODS] -s SERVICE -- COMMAND"
    echo "  -s SERVICE   K8s service, i.e. a pod selector (required)"
    echo "     COMMAND   Command to execute on the pods"
    echo "  -n NAMESPACE K8s namespace (optional)"
    echo "  -m MAX-PODS  Max number of pods to run on (optional; default=all)"
    echo "  -q           Quiet mode"
    echo "  -d           Dry run (don't actually exec)"
}

For example to run command curl http://google.com on all pods of a service with name s1 and namespace n1 , you need to execute ./kcdo -s s1 -n n1 -- curl http://google.com .

Answer 5

I needed access to all pods so I can change log level on a class so I did from the inside of one of the pods:

// Change level to DEBUG
host <service-name>| awk '{print $4}' | while read line; do
curl --location --request POST "http://$line:9111/actuator/loggers/com.foo.MyClassName" \
--header 'Content-Type: application/json' \
--data-raw '{"configuredLevel": "DEBUG"}' 
done
// Query level on all pods
host <service-name>| awk '{print $4}' | while read line; do    
curl --location --request GET "http://$line:9111/actuator/loggers/com.foo.MyClassName"
echo
done

You need host and curl to execute it.

Not sure if this is good practice.