Pod not communicating with service in GKE

Question

On a GKE cluster, I have client and server pods, with a client service and a server service.

My server service is:

apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    app: server-deployment
  ports:
    - port: 5000
      targetPort: 5000

When I access the client pod shell and run

nslookup server-cluster-ip-service

I get

Server:         server-IP
Address:        server-IP-Address

Name:   server-cluster-ip-service.default.svc.cluster.local
Address: IPAddress

** server can't find server-cluster-ip-service.svc.cluster.local: NXDOMAIN

** server can't find server-cluster-ip-service.cluster.local: NXDOMAIN


** server can't find server-cluster-ip-service.svc.cluster.local: NXDOMAIN
** server can't find server-cluster-ip-service.cluster.local: NXDOMAIN

** server can't find server-cluster-ip-service.us-central1-c.c.my-cluster.internal: NXDOMAIN

** server can't find server-cluster-ip-service.google.internal: NXDOMAIN

** server can't find server-cluster-ip-service.us-central1-c.c.my-cluster: NXDOMAIN

** server can't find server-cluster-ip-service.google.internal: NXDOMAIN

** server can't find server-cluster-ip-service.c.my-cluster.internal: NXDOMAIN
** server can't find server-cluster-ip-service.c.my-cluster.internal: NXDOMAIN

The service is running on port 5000, because when I set up a pod with busybox, I can curl from that pod like so:

curl server-cluster-ip-service:5000

And it returns json from my server.

After experimenting with what address to put in the fetch request in my client code, the only way I can get a 200 response is with this:

const getAllUsers = async () => {
    console.log("GETTING ALL USERS");
    const response = await fetch("server-cluster-ip-service.default.svc.cluster.local", {
  mode: 'cors',
  headers: {
    'Access-Control-Allow-Origin':'*'
  }
});
    const resp = await response
    console.log("RESPONSE", resp)
    const json = await response.json();

    setUsers(json);
  };

which returns not json and not apparently on port 5000,

whereas all attempts to query the service at port 5000 fail.

I have this running locally and it works fine. I have ruled out arm processor issues with my mac by building and pushing docker images in GKE from the cloud console. I am fairly confident this is a GKE issue, because the dns works locally, but why would it not work with GKE? I don't have any network policies I've set myself - could there be a node security group blocking access? I read about "shielding" as a node security policy configured at setup, but I don't know how to check if that's been configured?

Complete code below:

My server code is:

const express = require("express");
const bodyParser = require("body-parser");

var cors = require("cors");

const PORT = 5000;

const app = express();
app.use(cors());

app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());
app.use(express.static("public"));

app.listen(PORT, function () {
  console.log("listening on 5000");
});

app.get("/", (req, res) => {
  console.log("PROCESSING GET USERS REQUEST");
  const list = ["item1", "item2", "item3"];
  res.json(list);
});

My client code is:

import { useState, useEffect } from "react";

import "./editUser.css";

    function EditUser() {
      const [users, setUsers] = useState([]);
    
      const getAllUsers = async () => {
        console.log("GETTING ALL USERS");
        const response = await fetch("http://server-cluster-ip-service:5000");
        const json = await response.json();
    
        setUsers(json);
      };
    
      useEffect(() => {
        getAllUsers();
      }, []);
    
      return (
        <div className="App">
          <h1 data-position="header">Merr k8s testbed</h1>
    
          <section data-position="quotes">
            <h2>Console</h2>
            <ul>
              {users &&
                users.map((user) => (
                  <li>
                    <h3>{user}</h3>
                  </li>
                ))}
            </ul>
          </section>
        </div>
      );
    }
    
    export default EditUser;

My client-deployment.yaml is:

kind: Deployment
metadata:
  name: client-deployment
  labels:
    app: client-deployment
    component: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: client-deployment
  template:
    metadata: 
      labels:
        app: client-deployment
        component: web
    spec:
      containers:
        - name: client
          image: myDocker/k8s-client:latest
          ports:
            - containerPort: 3000

My server-deployment.yaml is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
  labels:
    app: server-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: server-deployment
  template:
    metadata:
      labels:
        app: server-deployment
    spec:
      containers:
        - name: server
          image: myDocker/k8s-server:latest
          ports:
            - containerPort: 5000

My client-cluster-ip-service.yaml is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
  labels:
    app: server-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: server-deployment
  template:
    metadata:
      labels:
        app: server-deployment
    spec:
      containers:
        - name: server
          image: myDocker/k8s-server:latest
          ports:
            - containerPort: 5000

My server-cluster-ip-service.yaml is:

apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    app: server-deployment
  ports:
    - port: 5000
      targetPort: 5000

Answer 1

I was able to see that you and jabbson concluded that probably the issue is with React. Just in case let me share with you that, a common root cause for this kind of issue is that the DNS inside Busybox does not work properly, due to the version (I cannot see in the screenshots and the code, the version that you are using). Most of the cases work with busybox images 1.28.4, empirically talking. You can try using that version.

You can use the following URL's thread as reference dns can't resolve kubernetes.default and/or cluster.local