i want to configure port forward 80->32181, 443->30598. 32181 and 30598 is NodePort of k8s ingress controller which i can establish connection correctly:
$ curl http://localhost:32181
<html>
<head><title>404 Not Found</title></head>
<body>
...
$ curl https://localhost:30598 -k
<html>
<head><title>404 Not Found</title></head>
<body>
...
What I have done is:
$ cat /proc/sys/net/ipv4/ip_forward
1
$ firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: cockpit dhcpv6-client frp http https kube-apiserver kube-kubelet ssh
ports:
protocols:
forward: no
masquerade: yes
forward-ports:
port=80:proto=tcp:toport=32181:toaddr=
port=443:proto=tcp:toport=30598:toaddr=
source-ports:
icmp-blocks:
rich rules:
but I cant access my nginx via 80 or 443:
$ curl https://localhost:443 -k
curl: (7) Failed to connect to localhost port 443: Connection refused
and more info:
centos: 8.2 4.18.0-348.2.1.el8_5.x86_64
k8s: 1.22(with calico(v3.21.0) network plugin)
firewalld: 0.9.3
and iptables output:
$ iptables -nvL -t nat --line-numbers
Chain PREROUTING (policy ACCEPT 51 packets, 2688 bytes)
num pkts bytes target prot opt in out source destination
1 51 2688 cali-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:6gwbT8clXdHdC1b1 */
2 51 2688 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
3 51 2688 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 50 packets, 2648 bytes)
num pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 1872 packets, 112K bytes)
num pkts bytes target prot opt in out source destination
1 1894 114K cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:O3lYWMrLQYEMJtB5 */
2 1862 112K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
3 0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 1922 packets, 116K bytes)
num pkts bytes target prot opt in out source destination
1 1894 114K cali-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:tVnHkvAo15HuiPy0 */
2 1911 115K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
3 758 45480 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain DOCKER (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-SERVICES (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-SVC-JD5MR3NA4I4DYORP tcp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
2 0 0 KUBE-SVC-Z6GDYMWE5TV2NNJN tcp -- * * 0.0.0.0/0 10.110.193.197 /* kubernetes-dashboard/dashboard-metrics-scraper cluster IP */ tcp dpt:8000
3 0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
4 0 0 KUBE-SVC-EDNDUDH2C75GIR6O tcp -- * * 0.0.0.0/0 10.97.201.174 /* ingress-nginx/ingress-nginx-controller:https cluster IP */ tcp dpt:443
5 0 0 KUBE-SVC-EZYNCFY2F7N6OQA2 tcp -- * * 0.0.0.0/0 10.103.242.141 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
6 0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
7 0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
8 0 0 KUBE-SVC-CEZPIJSAUFW5MYPQ tcp -- * * 0.0.0.0/0 10.97.166.112 /* kubernetes-dashboard/kubernetes-dashboard cluster IP */ tcp dpt:443
9 0 0 KUBE-SVC-H5K62VURUHBF7BRH tcp -- * * 0.0.0.0/0 10.104.154.95 /* lens-metrics/kube-state-metrics:metrics cluster IP */ tcp dpt:8080
10 0 0 KUBE-SVC-MOZMMOD3XZX35IET tcp -- * * 0.0.0.0/0 10.96.73.22 /* lens-metrics/prometheus:web cluster IP */ tcp dpt:80
11 0 0 KUBE-SVC-CG5I4G2RS3ZVWGLK tcp -- * * 0.0.0.0/0 10.97.201.174 /* ingress-nginx/ingress-nginx-controller:http cluster IP */ tcp dpt:80
12 1165 69528 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Chain KUBE-POSTROUTING (1 references)
num pkts bytes target prot opt in out source destination
1 1859 112K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000/0x4000
2 3 180 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK xor 0x4000
3 3 180 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ random-fully
Chain KUBE-MARK-DROP (0 references)
num pkts bytes target prot opt in out source destination
1 0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000
Chain KUBE-NODEPORTS (1 references)
num pkts bytes target prot opt in out source destination
1 2 120 KUBE-SVC-EDNDUDH2C75GIR6O tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https */ tcp dpt:30598
2 1 60 KUBE-SVC-CG5I4G2RS3ZVWGLK tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:http */ tcp dpt:32181
Chain KUBE-MARK-MASQ (27 references)
num pkts bytes target prot opt in out source destination
1 3 180 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000
Chain KUBE-SEP-IPE5TMLTCUYK646X (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.147 0.0.0.0/0 /* kube-system/kube-dns:metrics */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:192.168.103.147:9153
Chain KUBE-SEP-3LZLTHU4JT3FAVZK (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.149 0.0.0.0/0 /* kube-system/kube-dns:metrics */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:192.168.103.149:9153
Chain KUBE-SVC-JD5MR3NA4I4DYORP (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.96.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
2 0 0 KUBE-SEP-IPE5TMLTCUYK646X all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ statistic mode random probability 0.50000000000
3 0 0 KUBE-SEP-3LZLTHU4JT3FAVZK all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */
Chain KUBE-SEP-ZOAMCQDU54EOM4EJ (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.141 0.0.0.0/0 /* kubernetes-dashboard/dashboard-metrics-scraper */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes-dashboard/dashboard-metrics-scraper */ tcp to:192.168.103.141:8000
Chain KUBE-SVC-Z6GDYMWE5TV2NNJN (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.110.193.197 /* kubernetes-dashboard/dashboard-metrics-scraper cluster IP */ tcp dpt:8000
2 0 0 KUBE-SEP-ZOAMCQDU54EOM4EJ all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes-dashboard/dashboard-metrics-scraper */
Chain KUBE-SEP-HYE2IFAO6PORQFJR (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.0.176 0.0.0.0/0 /* default/kubernetes:https */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ tcp to:192.168.0.176:6443
Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
2 0 0 KUBE-SEP-HYE2IFAO6PORQFJR all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */
Chain KUBE-SEP-GJ4OJHBKIREWLMRS (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.146 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https */
2 2 120 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https */ tcp to:192.168.103.146:443
Chain KUBE-SVC-EDNDUDH2C75GIR6O (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.97.201.174 /* ingress-nginx/ingress-nginx-controller:https cluster IP */ tcp dpt:443
2 2 120 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https */ tcp dpt:30598
3 2 120 KUBE-SEP-GJ4OJHBKIREWLMRS all -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https */
Chain KUBE-SEP-K2CVHZPTBE2YAD6P (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.146 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */ tcp to:192.168.103.146:8443
Chain KUBE-SVC-EZYNCFY2F7N6OQA2 (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.103.242.141 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
2 0 0 KUBE-SEP-K2CVHZPTBE2YAD6P all -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */
Chain KUBE-SEP-S6VTWHFP6KEYRW5L (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.147 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.103.147:53
Chain KUBE-SEP-SFGZMYIS2CE4JD3K (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.149 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.103.149:53
Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
2 0 0 KUBE-SEP-S6VTWHFP6KEYRW5L all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ statistic mode random probability 0.50000000000
3 0 0 KUBE-SEP-SFGZMYIS2CE4JD3K all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
Chain KUBE-SEP-IJUMPPTQDLYXOX4B (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.147 0.0.0.0/0 /* kube-system/kube-dns:dns */
2 0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:192.168.103.147:53
Chain KUBE-SEP-C4W6TKYY5HHEG4RV (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.149 0.0.0.0/0 /* kube-system/kube-dns:dns */
2 0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:192.168.103.149:53
Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ udp -- * * !192.168.0.0/16 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
2 0 0 KUBE-SEP-IJUMPPTQDLYXOX4B all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ statistic mode random probability 0.50000000000
3 0 0 KUBE-SEP-C4W6TKYY5HHEG4RV all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */
Chain KUBE-SEP-GX372II3CQAGUHFM (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.145 0.0.0.0/0 /* kubernetes-dashboard/kubernetes-dashboard */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes-dashboard/kubernetes-dashboard */ tcp to:192.168.103.145:8443
Chain KUBE-SVC-CEZPIJSAUFW5MYPQ (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.97.166.112 /* kubernetes-dashboard/kubernetes-dashboard cluster IP */ tcp dpt:443
2 0 0 KUBE-SEP-GX372II3CQAGUHFM all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes-dashboard/kubernetes-dashboard */
Chain KUBE-SEP-I3RZS3REJP7POFLG (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.143 0.0.0.0/0 /* lens-metrics/kube-state-metrics:metrics */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* lens-metrics/kube-state-metrics:metrics */ tcp to:192.168.103.143:8080
Chain KUBE-SVC-H5K62VURUHBF7BRH (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.104.154.95 /* lens-metrics/kube-state-metrics:metrics cluster IP */ tcp dpt:8080
2 0 0 KUBE-SEP-I3RZS3REJP7POFLG all -- * * 0.0.0.0/0 0.0.0.0/0 /* lens-metrics/kube-state-metrics:metrics */
Chain KUBE-SEP-ROTMHDCXAI3T7IOR (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.144 0.0.0.0/0 /* lens-metrics/prometheus:web */
2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* lens-metrics/prometheus:web */ tcp to:192.168.103.144:9090
Chain KUBE-SVC-MOZMMOD3XZX35IET (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.96.73.22 /* lens-metrics/prometheus:web cluster IP */ tcp dpt:80
2 0 0 KUBE-SEP-ROTMHDCXAI3T7IOR all -- * * 0.0.0.0/0 0.0.0.0/0 /* lens-metrics/prometheus:web */
Chain KUBE-SEP-OAYGOO6JHJEB65WC (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ all -- * * 192.168.103.146 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:http */
2 1 60 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:http */ tcp to:192.168.103.146:80
Chain KUBE-SVC-CG5I4G2RS3ZVWGLK (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.97.201.174 /* ingress-nginx/ingress-nginx-controller:http cluster IP */ tcp dpt:80
2 1 60 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:http */ tcp dpt:32181
3 1 60 KUBE-SEP-OAYGOO6JHJEB65WC all -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:http */
Chain KUBE-PROXY-CANARY (0 references)
num pkts bytes target prot opt in out source destination
Chain cali-nat-outgoing (1 references)
num pkts bytes target prot opt in out source destination
1 49 3274 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:flqWnvo8yq4ULQLa */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst random-fully
Chain cali-POSTROUTING (1 references)
num pkts bytes target prot opt in out source destination
1 1894 114K cali-fip-snat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Z-c7XtVd2Bq7s_hA */
2 1894 114K cali-nat-outgoing all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:nYKhEzDlr11Jccal */
3 0 0 MASQUERADE all -- * tunl0 0.0.0.0/0 0.0.0.0/0 /* cali:SXWvdsbh4Mw7wOln */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL random-fully
Chain cali-PREROUTING (1 references)
num pkts bytes target prot opt in out source destination
1 51 2688 cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:r6XmIziWUJsdOK6Z */
Chain cali-fip-snat (1 references)
num pkts bytes target prot opt in out source destination
Chain cali-OUTPUT (1 references)
num pkts bytes target prot opt in out source destination
1 1894 114K cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:GBTAv2p5CwevEyJm */
Chain cali-fip-dnat (2 references)
num pkts bytes target prot opt in out source destination
Chain KUBE-KUBELET-CANARY (0 references)
num pkts bytes target prot opt in out source destination
To clarify I am posting Community Wiki answer.
The problem existed only during forwarding to a k8s service NodePort.
To solve the problem you have set up an External Nginx as a TCP Proxy.
Here one can find documentation about External NGINX .
Ingress does not directly support TCP services, so some additional configuration is necessary. Your NGINX Ingress Controller may have been deployed directly (ie with a Kubernetes spec file) or through the official Helm chart . The configuration of the TCP pass through will differ depending on the deployment approach.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.