繁体   English   中英

firewalld 端口转发到 k8s 节点端口不起作用

[英]firewalld port forward to k8s node port not working

我想配置端口转发 80->32181、443->30598。 32181 和 30598 是 k8s 入口 controller 的 NodePort,我可以正确建立连接:

$ curl http://localhost:32181
<html>
<head><title>404 Not Found</title></head>
<body>
...

$ curl https://localhost:30598 -k
<html>
<head><title>404 Not Found</title></head>
<body>
...

我所做的是:

$ cat /proc/sys/net/ipv4/ip_forward
1

$ firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: cockpit dhcpv6-client frp http https kube-apiserver kube-kubelet ssh
  ports:
  protocols:
  forward: no
  masquerade: yes
  forward-ports:
    port=80:proto=tcp:toport=32181:toaddr=
    port=443:proto=tcp:toport=30598:toaddr=
  source-ports:
  icmp-blocks:
  rich rules:

但我无法通过 80 或 443 访问我的 nginx:

$ curl https://localhost:443 -k
curl: (7) Failed to connect to localhost port 443: Connection refused

和更多信息:

centos:8.2 4.18.0-348.2.1.el8_5.x86_64

k8s:1.22(带有calico(v3.21.0)网络插件)

防火墙:0.9.3

和 iptables output:

$ iptables -nvL -t nat --line-numbers
Chain PREROUTING (policy ACCEPT 51 packets, 2688 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       51  2688 cali-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:6gwbT8clXdHdC1b1 */
2       51  2688 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
3       51  2688 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 50 packets, 2648 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 1872 packets, 112K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     1894  114K cali-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:O3lYWMrLQYEMJtB5 */
2     1862  112K KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
3        0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0

Chain OUTPUT (policy ACCEPT 1922 packets, 116K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     1894  114K cali-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:tVnHkvAo15HuiPy0 */
2     1911  115K KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
3      758 45480 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain DOCKER (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0

Chain KUBE-SERVICES (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-SVC-JD5MR3NA4I4DYORP  tcp  --  *      *       0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
2        0     0 KUBE-SVC-Z6GDYMWE5TV2NNJN  tcp  --  *      *       0.0.0.0/0            10.110.193.197       /* kubernetes-dashboard/dashboard-metrics-scraper cluster IP */ tcp dpt:8000
3        0     0 KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  *      *       0.0.0.0/0            10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
4        0     0 KUBE-SVC-EDNDUDH2C75GIR6O  tcp  --  *      *       0.0.0.0/0            10.97.201.174        /* ingress-nginx/ingress-nginx-controller:https cluster IP */ tcp dpt:443
5        0     0 KUBE-SVC-EZYNCFY2F7N6OQA2  tcp  --  *      *       0.0.0.0/0            10.103.242.141       /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
6        0     0 KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  *      *       0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
7        0     0 KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  *      *       0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
8        0     0 KUBE-SVC-CEZPIJSAUFW5MYPQ  tcp  --  *      *       0.0.0.0/0            10.97.166.112        /* kubernetes-dashboard/kubernetes-dashboard cluster IP */ tcp dpt:443
9        0     0 KUBE-SVC-H5K62VURUHBF7BRH  tcp  --  *      *       0.0.0.0/0            10.104.154.95        /* lens-metrics/kube-state-metrics:metrics cluster IP */ tcp dpt:8080
10       0     0 KUBE-SVC-MOZMMOD3XZX35IET  tcp  --  *      *       0.0.0.0/0            10.96.73.22          /* lens-metrics/prometheus:web cluster IP */ tcp dpt:80
11       0     0 KUBE-SVC-CG5I4G2RS3ZVWGLK  tcp  --  *      *       0.0.0.0/0            10.97.201.174        /* ingress-nginx/ingress-nginx-controller:http cluster IP */ tcp dpt:80
12    1165 69528 KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-POSTROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     1859  112K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x4000/0x4000
2        3   180 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK xor 0x4000
3        3   180 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ random-fully

Chain KUBE-MARK-DROP (0 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-NODEPORTS (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        2   120 KUBE-SVC-EDNDUDH2C75GIR6O  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https */ tcp dpt:30598
2        1    60 KUBE-SVC-CG5I4G2RS3ZVWGLK  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:http */ tcp dpt:32181

Chain KUBE-MARK-MASQ (27 references)
num   pkts bytes target     prot opt in     out     source               destination
1        3   180 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-SEP-IPE5TMLTCUYK646X (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.147      0.0.0.0/0            /* kube-system/kube-dns:metrics */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ tcp to:192.168.103.147:9153

Chain KUBE-SEP-3LZLTHU4JT3FAVZK (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.149      0.0.0.0/0            /* kube-system/kube-dns:metrics */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ tcp to:192.168.103.149:9153

Chain KUBE-SVC-JD5MR3NA4I4DYORP (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.96.0.10           /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
2        0     0 KUBE-SEP-IPE5TMLTCUYK646X  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ statistic mode random probability 0.50000000000
3        0     0 KUBE-SEP-3LZLTHU4JT3FAVZK  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */

Chain KUBE-SEP-ZOAMCQDU54EOM4EJ (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.141      0.0.0.0/0            /* kubernetes-dashboard/dashboard-metrics-scraper */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes-dashboard/dashboard-metrics-scraper */ tcp to:192.168.103.141:8000

Chain KUBE-SVC-Z6GDYMWE5TV2NNJN (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.110.193.197       /* kubernetes-dashboard/dashboard-metrics-scraper cluster IP */ tcp dpt:8000
2        0     0 KUBE-SEP-ZOAMCQDU54EOM4EJ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes-dashboard/dashboard-metrics-scraper */

Chain KUBE-SEP-HYE2IFAO6PORQFJR (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.0.176        0.0.0.0/0            /* default/kubernetes:https */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ tcp to:192.168.0.176:6443

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
2        0     0 KUBE-SEP-HYE2IFAO6PORQFJR  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */

Chain KUBE-SEP-GJ4OJHBKIREWLMRS (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.146      0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https */
2        2   120 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https */ tcp to:192.168.103.146:443

Chain KUBE-SVC-EDNDUDH2C75GIR6O (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.97.201.174        /* ingress-nginx/ingress-nginx-controller:https cluster IP */ tcp dpt:443
2        2   120 KUBE-MARK-MASQ  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https */ tcp dpt:30598
3        2   120 KUBE-SEP-GJ4OJHBKIREWLMRS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https */

Chain KUBE-SEP-K2CVHZPTBE2YAD6P (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.146      0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */ tcp to:192.168.103.146:8443

Chain KUBE-SVC-EZYNCFY2F7N6OQA2 (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.103.242.141       /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
2        0     0 KUBE-SEP-K2CVHZPTBE2YAD6P  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */

Chain KUBE-SEP-S6VTWHFP6KEYRW5L (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.147      0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.103.147:53

Chain KUBE-SEP-SFGZMYIS2CE4JD3K (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.149      0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.103.149:53

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
2        0     0 KUBE-SEP-S6VTWHFP6KEYRW5L  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ statistic mode random probability 0.50000000000
3        0     0 KUBE-SEP-SFGZMYIS2CE4JD3K  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SEP-IJUMPPTQDLYXOX4B (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.147      0.0.0.0/0            /* kube-system/kube-dns:dns */
2        0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.103.147:53

Chain KUBE-SEP-C4W6TKYY5HHEG4RV (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.149      0.0.0.0/0            /* kube-system/kube-dns:dns */
2        0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.103.149:53

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  udp  --  *      *      !192.168.0.0/16       10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
2        0     0 KUBE-SEP-IJUMPPTQDLYXOX4B  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ statistic mode random probability 0.50000000000
3        0     0 KUBE-SEP-C4W6TKYY5HHEG4RV  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */

Chain KUBE-SEP-GX372II3CQAGUHFM (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.145      0.0.0.0/0            /* kubernetes-dashboard/kubernetes-dashboard */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes-dashboard/kubernetes-dashboard */ tcp to:192.168.103.145:8443

Chain KUBE-SVC-CEZPIJSAUFW5MYPQ (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.97.166.112        /* kubernetes-dashboard/kubernetes-dashboard cluster IP */ tcp dpt:443
2        0     0 KUBE-SEP-GX372II3CQAGUHFM  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes-dashboard/kubernetes-dashboard */

Chain KUBE-SEP-I3RZS3REJP7POFLG (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.143      0.0.0.0/0            /* lens-metrics/kube-state-metrics:metrics */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* lens-metrics/kube-state-metrics:metrics */ tcp to:192.168.103.143:8080

Chain KUBE-SVC-H5K62VURUHBF7BRH (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.104.154.95        /* lens-metrics/kube-state-metrics:metrics cluster IP */ tcp dpt:8080
2        0     0 KUBE-SEP-I3RZS3REJP7POFLG  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* lens-metrics/kube-state-metrics:metrics */

Chain KUBE-SEP-ROTMHDCXAI3T7IOR (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.144      0.0.0.0/0            /* lens-metrics/prometheus:web */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* lens-metrics/prometheus:web */ tcp to:192.168.103.144:9090

Chain KUBE-SVC-MOZMMOD3XZX35IET (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.96.73.22          /* lens-metrics/prometheus:web cluster IP */ tcp dpt:80
2        0     0 KUBE-SEP-ROTMHDCXAI3T7IOR  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* lens-metrics/prometheus:web */

Chain KUBE-SEP-OAYGOO6JHJEB65WC (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.146      0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:http */
2        1    60 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:http */ tcp to:192.168.103.146:80

Chain KUBE-SVC-CG5I4G2RS3ZVWGLK (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.97.201.174        /* ingress-nginx/ingress-nginx-controller:http cluster IP */ tcp dpt:80
2        1    60 KUBE-MARK-MASQ  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:http */ tcp dpt:32181
3        1    60 KUBE-SEP-OAYGOO6JHJEB65WC  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:http */

Chain KUBE-PROXY-CANARY (0 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain cali-nat-outgoing (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       49  3274 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:flqWnvo8yq4ULQLa */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst random-fully

Chain cali-POSTROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     1894  114K cali-fip-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Z-c7XtVd2Bq7s_hA */
2     1894  114K cali-nat-outgoing  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:nYKhEzDlr11Jccal */
3        0     0 MASQUERADE  all  --  *      tunl0   0.0.0.0/0            0.0.0.0/0            /* cali:SXWvdsbh4Mw7wOln */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL random-fully

Chain cali-PREROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       51  2688 cali-fip-dnat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:r6XmIziWUJsdOK6Z */

Chain cali-fip-snat (1 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain cali-OUTPUT (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     1894  114K cali-fip-dnat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:GBTAv2p5CwevEyJm */

Chain cali-fip-dnat (2 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain KUBE-KUBELET-CANARY (0 references)
num   pkts bytes target     prot opt in     out     source               destination

为了澄清我正在发布社区 Wiki 答案。

该问题仅在转发到 k8s 服务 NodePort 期间存在。

为了解决这个问题,您设置了一个外部 Nginx 作为 TCP 代理。

在这里可以找到有关External NGINX文档

Ingress 不直接支持 TCP 服务,所以需要一些额外的配置。 您的 NGINX 入口 Controller 可能已直接部署(即使用 Kubernetes 规范文件)或通过官方 Helm 图表部署。 TCP 直通的配置因部署方法而异。

暂无
暂无

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM