Zabbix 6.0 Agent doesn't collect metrics in Kubernetes and can't talk with server

Question

I've been setting up Zabbix Agent in kubernetes the past few days and I'm stuck on what I think is the most basic thing. I've created a brand new namespace, setup MySQL there and configured it, has the zabbixUser, pass and everything, I've setup Zabbix Server from this Helm Chart.

Here are the values files for it

zabbixserver:
  image:
    #repository: zabbix/zabbix-server-pgsql
    repository: "zabbix/zabbix-server-mysql" #https://hub.docker.com/r/zabbix/zabbix-server-pgsql
    tag: "ubuntu-6.0.3"

  DB_SERVER_HOST: "zabbix-mysql"
  DB_SERVER_PORT: "3306"

  # -- Extra environment variables. A list of additional environment variables. See example: https://github.com/cetic/helm-zabbix/blob/master/docs/example/kind/values.yaml
  #  taken from mysql config
  extraEnv:
    - name: "MYSQL_USER"
      value: "zabbixUser"
    - name: "MYSQL_PASSWORD"
      value: "zabbixPass"
    - name: "MYSQL_ROOT_PASSWORD"
      value: "dyEMnUK69d"
    - name: "MYSQL_DATABASE"
      value: "zabbix"

postgresql:
  enabled: false

zabbixproxy:
  enabled: false
    #  # The variable is used to specify debug level, from 0 to 5
    #  ZBX_DEBUGLEVEL: 3
    #  ZBX_PROXYMODE: 1
    #  ZBX_SERVER_HOST: 10.190.8.122

# Installs zabbix agent next on the same pod as server
zabbixagent:
  enabled: true
  env:
    - name: ZBX_DEBUGLEVEL
      value: 4


zabbixweb:
  image:
    repository: "zabbix/zabbix-web-nginx-mysql" #https://hub.docker.com/r/zabbix/zabbix-web-nginx-mysql
    #repository: zabbix/zabbix-web-nginx-psql #https://hub.docker.com/r/zabbix/zabbix-web-nginx-pgsql
    tag: "ubuntu-6.0.3"

# kubectl get services
  ZBX_SERVER_HOST: zabbix-server-zabbix-server
  DB_SERVER_HOST: zabbix-mysql
  DB_SERVER_PORT: 3306

  extraEnv:
    - name: "MYSQL_USER"
      value: "zabbixUser"
    - name: "MYSQL_PASSWORD"
      value: "zabbixPass"
    - name: "MYSQL_ROOT_PASSWORD"
      value: "dyEMnUK69d"
    - name: "MYSQL_DATABASE"
      value: "zabbix"

ingress:
  enabled: true
  hosts:
    - host: better-zabbix.genesis
      paths:
        - path: /
          pathType: Prefix
  ingressClassName: nginx-internal

NOTE: It's hosted in my company's internal domain, and MySQL is also for testing purposes so no prod credentials.

I configured DNS as well so it all works as you can see.

Now I wanted to set up the Zabbix agent fromthis official helm chart.

Here is also a useful link showing ready templates to help get metrics from the GKE cluster - however, note that when you set up the Zabbix server from helm chart, he comes with them built-in. So I guess it's useful if you've got an older version and are on VM.

Here are my values.yaml for agent

zabbixAgent:
  image:
    repository: zabbix/zabbix-agent
    tag: alpine-6.0.3
  env:
    - name: ZBX_SERVER_HOST
      value: 10.190.40.58 # after zabbix server was created, this is the IP of his pod
      #value: 127.0.0.1
      #value: "better-zabbix.genesis"

zabbixProxy:
  enabled: false
rbac:
  create: false
serviceAccount:
  create: false

And now we get to the problem I have

Every 2.0s: kubectl get pods -o wide                                                                                                       colgate.local: Sat Apr 16 15:54:05 2022

NAME                                               READY   STATUS    RESTARTS   AGE     IP             NODE                                 NOMINATED NODE   READINESS GATES
potato                                             1/1     Running   0          5h26m   10.190.4.102   gke-tools-single-nat-35d661ab-sf1c   <none>           <none>
zabbix-agent-7d4xv                                 1/1     Running   0          3h49m   10.222.0.59    gke-tools-single-nat-d4ec251a-hp7k   <none>           <none>
zabbix-agent-856lm                                 1/1     Running   0          3h46m   10.222.0.39    gke-tools-main-tools-bc42e123-6fpq   <none>           <none>
zabbix-agent-c5xtp                                 1/1     Running   0          3h47m   10.222.0.9     gke-tools-main-tools-bc42e123-ycgx   <none>           <none>
zabbix-agent-dn78k                                 1/1     Running   0          3h49m   10.222.0.98    gke-tools-single-nat-d4ec251a-i0a1   <none>           <none>
zabbix-agent-fbpzn                                 1/1     Running   0          3h49m   10.222.0.54    gke-tools-single-nat-d4ec251a-kxwn   <none>           <none>
zabbix-agent-gsvgt                                 1/1     Running   0          3h47m   10.222.0.51    gke-tools-main-tools-bc42e123-oc1i   <none>           <none>
zabbix-agent-h4bkn                                 1/1     Running   0          3h49m   10.222.0.50    gke-tools-single-nat-2dc0d382-1uv1   <none>           <none>
zabbix-agent-kube-state-metrics-5968c76968-6jb98   1/1     Running   0          3h59m   10.190.1.48    gke-tools-single-nat-d4ec251a-i0a1   <none>           <none>
zabbix-agent-pfs5s                                 1/1     Running   0          3h47m   10.222.0.55    gke-tools-single-nat-d4ec251a-4o43   <none>           <none>
zabbix-agent-plk7s                                 1/1     Running   0          3h46m   10.222.0.49    gke-tools-single-nat-35d661ab-xxsw   <none>           <none>
zabbix-agent-qm5c5                                 1/1     Running   0          3h48m   10.222.0.41    gke-tools-single-nat-35d661ab-m4ct   <none>           <none>
zabbix-agent-wk59p                                 1/1     Running   0          3h48m   10.222.0.5     gke-tools-single-nat-2dc0d382-zp2s   <none>           <none>
zabbix-agent-wl97k                                 1/1     Running   0          3h48m   10.222.0.88    gke-tools-main-tools-89255868-mxfl   <none>           <none>
zabbix-agent-z6vxh                                 1/1     Running   0          3h49m   10.222.0.46    gke-tools-single-nat-35d661ab-sf1c   <none>           <none>
zabbix-mysql-0                                     1/1     Running   0          3h59m   10.190.1.49    gke-tools-single-nat-d4ec251a-i0a1   <none>           <none>
zabbix-mysql-client                                1/1     Running   0          5h7m    10.190.0.215   gke-tools-single-nat-2dc0d382-zp2s   <none>           <none>
zabbix-server-zabbix-server-0                      2/2     Running   0          3h59m   10.190.2.99    gke-tools-single-nat-2dc0d382-1uv1   <none>           <none>
zabbix-server-zabbix-web-665c9b767-ssdt6           1/1     Running   0          3h59m   10.190.4.113   gke-tools-single-nat-35d661ab-sf1c   <none>           <none>

Logs from zabbix-agent pods are like this

3438519:20220416:135248.449 failed to accept an incoming connection: connection from "10.222.0.54" rejected, allowed hosts: "10.190.40.58"

So what does it mean? In zabbix-agent values.yaml I wrote that zabbix server host is

    - name: ZBX_SERVER_HOST
      value: 10.190.40.58

but the zabbix-agent says literally its own IP cannot be whitelisted? what does it all mean?

also, these are the logs from zabbix server

   208:20220416:135600.044 [ Kubernetes discovery ] ERROR: Incorrect JSON. Check debug log for more information.
   208:20220416:135600.045 [ Kubernetes discovery ] ERROR: Incorrect JSON. Check debug log for more information.
   226:20220416:135600.045 [ Kubernetes ] ERROR: Error: cannot get URL: Couldn't connect to server.
   208:20220416:135600.047 [ Kubernetes discovery ] ERROR: Incorrect JSON. Check debug log for more information.
   208:20220416:135600.048 [ Kubernetes discovery ] ERROR: Incorrect JSON. Check debug log for more information.
   226:20220416:135600.049 [ Kubernetes ] ERROR: Error: cannot get URL: Couldn't connect to server.
   208:20220416:135600.051 [ Kubernetes discovery ] ERROR: Incorrect JSON. Check debug log for more information.
   208:20220416:135600.052 [ Kubernetes discovery ] ERROR: Incorrect JSON. Check debug log for more information.

The autoregistration I have set up on zabbix-server works

and we can see the hosts

but the problem stays that agents can't get metrics/data from nodes

I'll add that I configured the template with token

As it was written in this instructions in Zabbix Setup at the bottom of the page. Also, the one more unknown to me is the Zabbix configuration part from here All these {$KUBE.API.ENDPOINT.URL} what should they point to?

You guys would make my day by giving any hint as, why nodes are not seen, why agents apparently can't talk to the server, and whether should I change the {$KUBE.API.ENDPOINT.URL} or leave it as it was with default values.

Answer 1

{$KUBE.API.ENDPOINT.URL} this should point to https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT .

I get the values from a running pod. However, I did not try as a env value in Zabbix.