How to check kubelet readiness before running kubectl port-forward?

Question

I wrote a script that starts minikube and uses kubectl port-forward to forward a local port to one of the services running in the kubernetes cluster. It runs kubectl port-forward in the background so it can run a web-server on localhost with access to a service in the cluster.

There's a race-condition where kubectl port-forward fails because the kubelet isn't ready to accept connections yet.

error upgrading connection: error dialing backend: dial tcp 127.0.1.1:10250: getsockopt: connection refused

Short of putting a sleep in my script, how might I check whether the kubelet is ready before running the port-forward?

Since the error message indicates we expect the kubelet to be listening for tcp connections on port 10250, I thought I could do something like this:

ss --listening --tcp --numeric --processes \
  '( sport = 10250 )' \
  | grep --quiet kubelet

But I found that even once the kubelet is ready, that ss command doesn't find it. I don't know if this is due to the 127.0.1.1 ip address or if there's something else I'm missing.

The main script, run-server , looks like:

#! /usr/bin/env bash

LOCAL_PORT=5678

cleanup() {
    local pid=$(ss --listening --tcp --processes \
                   "( sport = ${LOCAL_PORT} )" \
                | grep -oP 'pid=\K\d+')

    [ -n "$pid" ] && kill "$pid"
}
trap 'cleanup' EXIT

minikube start
run-port-forward $LOCAL_PORT &
FOO_SERVICE_URL=localhost:${LOCAL_PORT} bin/rails server

and run-port-forward looks like

#! /usr/bin/env bash

LOCAL_PORT=$1

pod=$(kubectl --context=minikube get pod \
      | grep -m 1 -oP "foo-service\S*")

port=$(kubectl --context=minikube get pod ${pod} -o json \
       | jq -r .spec.containers[0].ports[0].containerPort)

kubectl --context=minikube port-forward \
    ${pod} ${LOCAL_PORT}:${port}

Answer 1

Though I haven't yet found a way to check for kubelet readiness using kubectl, I came up with a workaround that performs a few retries.

This is the first time I've used recursion in a bash script. I got a kick out of that.

try_port_forward() {
    retries=$1

    if (( retries > 0 )); then
        sleep 1
        kubectl --context=minikube port-forward \
            ${pod} ${LOCAL_PORT}:${port} \
            2>/dev/null ||
            try_port_forward $(( retries - 1 ))
    fi
}

try_port_forward 10

Answer 2

But I found that even once the kubelet is ready, that ss command doesn't find it. I don't know if this is due to the 127.0.1.1 ip address or if there's something else I'm missing.

minikube runs on a VM on your machine which has its own interface and IP address. You can retrieve it using minikube ip

So if you did:

curl -X GET http://$(minikube ip):10250/healthz

Assuming you've configured the following options :

--healthz-bind-address 0.0.0.0
The IP address for the healthz server to serve on (set to 0.0.0.0 for all IPv4 interfaces and `::` for all IPv6 interfaces) (default 127.0.0.1)
--healthz-port int32
The port of the localhost healthz endpoint (set to 0 to disable) (default 10248)