Kubernetes Pod/容器联网

Question

我在与 Pod 中的一个容器联网时遇到问题。 我创建了kubernetes集群（一主一节点）供学习。 Master 安装在一台 VM 上，Node 安装在第二台 VM 上。

节点列表：

NAME      STATUS   ROLES                  AGE   VERSION
kmaster   Ready    control-plane,master   68m   v1.20.2
ubuntu    Ready    <none>                 57m   v1.20.2

集群信息：

Kubernetes control plane is running at https://192.168.0.102:6443
KubeDNS is running at https://192.168.0.102:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Everything was working fine until VM's restarting.
One Pod with eclipse-mosquitto mqtt broker has some problem with networking with container inside.

下面的 pod 列表：

NAME                          READY   STATUS    RESTARTS   AGE   IP                NODE     NOMINATED NODE   READINESS GATES
mqttbroker-574b696f76-x26jq   1/1     Running   0          57m   192.168.243.193   ubuntu   <none>           <none>

下面对有问题的 pod 进行描述：

Name:         mqttbroker-574b696f76-x26jq
Namespace:    default
Priority:     0
Node:         ubuntu/192.168.0.121
Start Time:   Mon, 08 Feb 2021 00:18:16 -0800
Labels:       app=mqttbroker
              pod-template-hash=574b696f76
Annotations:  cni.projectcalico.org/podIP: 192.168.243.193/32
              cni.projectcalico.org/podIPs: 192.168.243.193/32
Status:       Running
IP:           192.168.243.193
IPs:
  IP:           192.168.243.193
Controlled By:  ReplicaSet/mqttbroker-574b696f76
Containers:
  mosquitto:
    Container ID:   docker://73417f8046abfd8773154cde02b7366f9d7c26331b03b141e89855deec71e273
    Image:          eclipse-mosquitto
    Image ID:       docker-pullable://eclipse-mosquitto@sha256:0916ce445d7f3945168966ab987515a081f2f2c761acb85ae00ff44d97d3a84f
    Port:           1883/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 08 Feb 2021 00:18:20 -0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     750m
      memory:  700Mi
    Requests:
      cpu:        500m
      memory:     500Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qgs7v (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-qgs7v:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qgs7v
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

但是，当我检查容器日志时，分配 IP 时出现一些错误：容器日志

1612772300: mosquitto version 2.0.7 starting
1612772300: Config loaded from /mosquitto/config/mosquitto.conf.
1612772300: Starting in local only mode. Connections will only be possible from clients running on this machine.
1612772300: Create a configuration file which defines a listener to allow remote access.
1612772300: Opening ipv4 listen socket on port 1883.
1612772300: Opening ipv6 listen socket on port 1883.
1612772300: Error: Address not available
1612772300: mosquitto version 2.0.7 running

当我检查分配给容器的地址时，我看到了奇怪的 IP 掩码。 我想这可能是我的问题。 下面 ifconfig output。

eth0      Link encap:Ethernet  HWaddr 5A:27:37:13:F7:06  
          inet addr:192.168.243.193  Bcast:192.168.243.193  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:14328 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14315 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:927289 (905.5 KiB)  TX bytes:773046 (754.9 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:138 errors:0 dropped:0 overruns:0 frame:0
          TX packets:138 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:7597 (7.4 KiB)  TX bytes:7597 (7.4 KiB)

下面的部署和服务：

kind: Deployment
apiVersion: apps/v1
metadata:
  name: mqttbroker
  labels:
    app: mqttbroker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mqttbroker
  template:
    metadata:
      labels:
        app: mqttbroker
    spec:
      containers:
      - name: mosquitto
        image: eclipse-mosquitto
        resources:
          requests:
            cpu: "0.5"
            memory: "500Mi"
          limits:
            cpu: "0.75"
            memory: "700Mi"
        ports:
          - name: mqttbroker-1883
            containerPort: 1883

集群 IP 服务

apiVersion: v1  
kind: Service  
metadata:  
  name: mosquitto-service  
spec:
  type: ClusterIP
  selector:  
    app: mqttbroker  
  ports:
    - name: service-mqtt-1883
      protocol: TCP  
      port: 1883
      targetPort: 1883

节点端口 IP 服务

apiVersion: v1
kind: Service
metadata:
  name: mosquittoservice
spec:
  type: NodePort
  ports:
  - name: "1883"
    port: 80
    targetPort:
    nodePort: 30081 # acces service via external port number
  selector:
    app: mqttbroker

服务清单：

NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE    SELECTOR
kubernetes          ClusterIP   10.96.0.1        <none>        443/TCP        116m   <none>
mosquitto-service   ClusterIP   10.98.70.62      <none>        1883/TCP       101m   app=mqttbroker
mosquittoservice    NodePort    10.106.207.192   <none>        80:30081/TCP   101m   app=mqttbroker

蚊子服务的描述

Name:              mosquitto-service
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=mqttbroker
Type:              ClusterIP
IP Families:       <none>
IP:                10.98.70.62
IPs:               10.98.70.62
Port:              service-mqtt-1883  1883/TCP
TargetPort:        1883/TCP
Endpoints:         192.168.243.193:1883
Session Affinity:  None
Events:            <none>

蚊子服务的描述

Name:                     mosquittoservice
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=mqttbroker
Type:                     NodePort
IP Families:              <none>
IP:                       10.106.207.192
IPs:                      10.106.207.192
Port:                     1883  80/TCP
TargetPort:               1883/TCP
NodePort:                 1883  30081/TCP
Endpoints:                192.168.243.193:1883
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

我的问题是什么可能导致这个问题以及如何解决它？

谢谢你的答案！！

编辑：当我调用容器的 function 时，如下所示工作正常。

kubectl exec -it mqttbroker-574b696f76-x26jq -- mositto_sub -h localhost -p 1883 -t topic -C 4

如果我将“localhost”更改为 IP 地址，我会收到连接被拒绝。

kubectl exec -it mqttbroker-574b696f76-x26jq -- mositto_sub -h 192.168.0.102 -p 1883 -t topic -C 4
Error: Connection refused
command terminated with exit code 1

如果我像下面那样调用 function，我也会被拒绝连接。

kubectl exec -it mqttbroker-574b696f76-x26jq -- mosquitto_sub -h 192.168.0.102 -p 30081 -t topic -C 4
Error: Connection refused
command terminated with exit code 1

Answer 1

问题和解决方案在评论部分确定。

问题的根本原因是在1883上缺少listener器配置。

类似问题的其他可能解决方案可以在 Github Thread- Mosquitto 无法启动中找到。

问题已通过使用ConfigMap解决，其中指定了1883上的listener器。

当我将 mosquitto.conf 文件挂载为 ConfigMap 时，我在其中指定了“listener 1883”，我的代理工作正常，其他订阅者获取数据。