从另一个 GKE 私有集群访问私有 GKE 集群中的节点端口服务

Question

I am using Google cloud and I have two GKE private clusters.

One of them contains some services installed as nodePort. The other cluster needs to connect to this one and access the services exposed.

The cluster with services exposed has only one node with a private IP. I can ping this node with success from the another cluster using this private IP.

But how can I access the services?

I also tried to config some firewall rules with no success.

Answer 1

Please take a look at below example which shows how to make a connection to a service ( NodePort ) between two private GKE clusters:

This example will use two GKE clusters:

• gke-private-cluster-main - this will be the cluster with a simple hello-app
• gke-private-cluster-europe - this cluster will be able to communicate with the main cluster

To simplify it all the clusters will have one node only.

Create a deployment and a service on gke-private-cluster-main

Below is a simple example of hello-app and a service which will expose hello-app on port 30051 :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  selector:
    matchLabels:
      app: hello
      version: 1.0.0
  replicas: 3
  template:
    metadata:
      labels:
        app: hello
        version: 1.0.0
    spec:
      containers:
      - name: hello
        image: "gcr.io/google-samples/hello-app:1.0"
        env:
        - name: "PORT"
          value: "50001"
---
apiVersion: v1
kind: Service
metadata:
  name: hello-service
spec:
  selector:
    app: hello
  ports:
    - name: hello-port
      port: 50001
      targetPort: 50001
      nodePort: 30051
  type: NodePort

Apply it and check the internal IP address of a node which spawned this pods. You can check it with either:

• GCP -> Compute Engine -> VM Instances
• kubectl get nodes -o wide

In my case it was 10.156.0.2

Try to access it from gke-private-cluster-europe

You can SSH to a node of gke-private-cluster-europe and try to invoke command from a node: curl 10.156.0.2:30051 . You should be able to communicate with this service and get the output like below:

Hello, world!
Version: 1.0.0
Hostname: hello-5d79ccdb55-vrrbs

To check connectivity from the inside of a pod you would need a image that has already curl builtin. Internet is a place for all kinds of awesome things and in fact there is a image that has curl available. You can spawn a pod with curl with below YAML :

apiVersion: v1
kind: Pod
metadata:
  name: curl
  namespace: default
spec:
  containers:
  - image: curlimages/curl
    command:
      - sleep
      - "infinity"
    imagePullPolicy: IfNotPresent
    name: curl
  restartPolicy: Always

After applying above YAML , you can exec into the pod and check for yourself with below commands:

• $ kubectl exec -it curl -- /bin/sh
• $ curl 10.156.0.2:30051

The output from the inside of a cluster will look like this:

curl: (28) Failed to connect to 10.156.0.2 port 30051: Operation timed out

It worked for a node but it does not work for a pod.

Allow the traffic:

To allow above network connectivity you will need to:

• Open Google Cloud Platform
  • Check the network tag of a gke-private-cluster-main node
    • Go to Compute Engine
    • Find the node of a gke-private-cluster-main
    • Click on it to get more details
    • Copy the network tag which should look similar to: gke-gke-private-cluster-main-80fe50b2-node
  • Check the pod address range of gke-private-cluster-europe :
    • Go to Kubernetes Engine
    • Find your gke-private-cluster-europe
    • Click on it to get more details
    • Copy the pod address range which should look similar to: 10.24.0.0/14

With network tag and pod range copied you can create your firewall rule. Please go to:

VPC Network -> Firewall rules -> Create a firewall rule

Please take a specific look on parts where network tag and pod range ip is used as it will be different for you.

Apply it and check again if a pod in gke-private-cluster-europe can access the 10.156.0.2:30051 .

It should give you output below:

Hello, world!
Version: 1.0.0
Hostname: hello-5d79ccdb55-6s8xh

Please let me know if you have any questions in that.

Answer 2

not supported by google, only directly attached VPC can access GKE on that VPC https://issuetracker.google.com/issues/244483997

Connecting to Private GKE cluster using 3rd party VPN