kube-proxy：缺少服务的iptables-DNAT规则

Question

kube-proxy doesn't create DNAT-rules on nodes for services registered on master. everything works fine beside this service-ip-to-pod-ip NATing.

my setup:

kubernetes master: 10.98.99.176/24 (running: api-srv, scheduler, controller-manager)  
kubernetes node1: 10.98.99.136/24 with CIDR 10.116.0.0/24 (running: kubelet, kube-proxy)  
kubernetes node2: 10.98.99.137/24 with CIDR 10.116.1.0/24 (running: kubelet, kube-proxy)  

CIDRs are configured on master via Node.spec.PodCIDR

cbr0-bridge is created on kubelet start.
routing is set up an works.
all fine.

i can ping the kube-dns-container running on node1 from a node2 container via its container ip (10.116.0.2) but not via its service ip (10.0.0.10). i see this icmp-packets moving out on eth0 to default-gw (10.116.1.2 -> 10.0.0.10)

so when i have a look at iptables-save output:

# Generated by iptables-save v1.4.21 on Fri Oct 28 14:08:56 2016
*filter
:INPUT ACCEPT [40:11192]
:FORWARD ACCEPT [3:180]
:OUTPUT ACCEPT [77:31363]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -j KUBE-FIREWALL
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-SERVICES -d 10.0.0.10/32 -p udp -m comment --comment "default/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.0.0.10/32 -p tcp -m comment --comment "default/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Fri Oct 28 14:08:56 2016
# Generated by iptables-save v1.4.21 on Fri Oct 28 14:08:56 2016
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-SM34KKATJ2TS55C5 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-D376NYSDDVFPF2KN - [0:0]
:KUBE-SVC-N6R7PS4OMIK6NEO2 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING ! -d 10.0.0.0/8 -m addrtype ! --dst-type LOCAL -j MASQUERADE
-A POSTROUTING ! -d 10.0.0.0/8 -m addrtype ! --dst-type LOCAL -j MASQUERADE
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-SM34KKATJ2TS55C5 -s 10.98.99.176/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-SM34KKATJ2TS55C5 -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-SM34KKATJ2TS55C5 --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.98.99.176:6443
-A KUBE-SERVICES -d 10.0.0.10/32 -p udp -m comment --comment "default/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-D376NYSDDVFPF2KN
-A KUBE-SERVICES -d 10.0.0.10/32 -p tcp -m comment --comment "default/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-N6R7PS4OMIK6NEO2
-A KUBE-SERVICES -d 10.0.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 180 --reap --name KUBE-SEP-SM34KKATJ2TS55C5 --mask 255.255.255.255 --rsource -j KUBE-SEP-SM34KKATJ2TS55C5
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-SM34KKATJ2TS55C5
COMMIT
# Completed on Fri Oct 28 14:08:56 2016

there are DNAT-rules missing for something like 10.0.0.10 -> 10.116.0.2, isn't it?

here the kube-proxy -debug-log-output while starting:

systemd[1]: Stopping Kubernetes Kube-Proxy Server...
systemd[1]: Starting Kubernetes Kube-Proxy Server...
systemd[1]: Started Kubernetes Kube-Proxy Server.
kube-proxy[29691]: I1028 13:37:36.485645   29691 server.go:155] setting OOM scores is unsupported in this build
kube-proxy[29691]: I1028 13:37:36.487545   29691 server.go:202] Using iptables Proxier.
kube-proxy[29691]: I1028 13:37:36.491592   29691 server.go:214] Tearing down userspace rules.
kube-proxy[29691]: I1028 13:37:36.491628   29691 iptables.go:380] running iptables -C [OUTPUT -t nat -m comment --comment handle ClusterIPs; NOTE: this must be before the NodePort rules -j KUBE-PORTALS-HOST]
kube-proxy[29691]: I1028 13:37:36.492721   29691 iptables.go:380] running iptables -C [PREROUTING -t nat -m comment --comment handle ClusterIPs; NOTE: this must be before the NodePort rules -j KUBE-PORTALS-CONTAINER]
kube-proxy[29691]: I1028 13:37:36.493751   29691 iptables.go:380] running iptables -C [OUTPUT -t nat -m addrtype --dst-type LOCAL -m comment --comment handle service NodePorts; NOTE: this must be the last rule in the chain -j KUBE-NODEPORT-HOST]
kube-proxy[29691]: I1028 13:37:36.494708   29691 iptables.go:380] running iptables -C [PREROUTING -t nat -m addrtype --dst-type LOCAL -m comment --comment handle service NodePorts; NOTE: this must be the last rule in the chain -j KUBE-NODEPORT-CONTAINER]
kube-proxy[29691]: I1028 13:37:36.495659   29691 iptables.go:380] running iptables -C [INPUT -t filter -m comment --comment Ensure that non-local NodePort traffic can flow -j KUBE-NODEPORT-NON-LOCAL]
kube-proxy[29691]: I1028 13:37:36.496493   29691 iptables.go:380] running iptables -F [KUBE-PORTALS-CONTAINER -t nat]
kube-proxy[29691]: I1028 13:37:36.497316   29691 iptables.go:380] running iptables -F [KUBE-PORTALS-HOST -t nat]
kube-proxy[29691]: I1028 13:37:36.498291   29691 iptables.go:380] running iptables -F [KUBE-NODEPORT-HOST -t nat]
kube-proxy[29691]: I1028 13:37:36.499014   29691 iptables.go:380] running iptables -F [KUBE-NODEPORT-CONTAINER -t nat]
kube-proxy[29691]: I1028 13:37:36.500045   29691 iptables.go:380] running iptables -F [KUBE-NODEPORT-NON-LOCAL -t filter]
kube-proxy[29691]: I1028 13:37:36.500861   29691 reflector.go:202] Starting reflector *api.Service (15m0s) from pkg/proxy/config/api.go:30
kube-proxy[29691]: I1028 13:37:36.500897   29691 reflector.go:202] Starting reflector *api.Endpoints (15m0s) from pkg/proxy/config/api.go:33
kube-proxy[29691]: I1028 13:37:36.500985   29691 conntrack.go:40] Setting nf_conntrack_max to 131072
kube-proxy[29691]: I1028 13:37:36.501009   29691 reflector.go:253] Listing and watching *api.Service from pkg/proxy/config/api.go:30
kube-proxy[29691]: I1028 13:37:36.501233   29691 reflector.go:253] Listing and watching *api.Endpoints from pkg/proxy/config/api.go:33
kube-proxy[29691]: I1028 13:37:36.501372   29691 conntrack.go:57] Setting conntrack hashsize to 32768
kube-proxy[29691]: I1028 13:37:36.502921   29691 config.go:208] Calling handler.OnServiceUpdate()
kube-proxy[29691]: I1028 13:37:36.502938   29691 proxier.go:440] Adding new service "default/kube-dns:dns" at 10.0.0.10:53/UDP
kube-proxy[29691]: I1028 13:37:36.503016   29691 proxier.go:453] added serviceInfo(default/kube-dns:dns): (*iptables.serviceInfo)(0xc82049b9a0)({
kube-proxy[29691]: clusterIP: (net.IP) (len=16 cap=16) 10.0.0.10,
kube-proxy[29691]: port: (int) 53,
kube-proxy[29691]: protocol: (api.Protocol) (len=3) "UDP",
kube-proxy[29691]: nodePort: (int) 0,
kube-proxy[29691]: loadBalancerStatus: (api.LoadBalancerStatus) {
kube-proxy[29691]: Ingress: ([]api.LoadBalancerIngress) {
kube-proxy[29691]: }
kube-proxy[29691]: },
kube-proxy[29691]: sessionAffinityType: (api.ServiceAffinity) (len=4) "None",
kube-proxy[29691]: stickyMaxAgeSeconds: (int) 180,
kube-proxy[29691]: externalIPs: ([]string) <nil>,
kube-proxy[29691]: loadBalancerSourceRanges: ([]string) <nil>
kube-proxy[29691]: })
kube-proxy[29691]: I1028 13:37:36.503030   29691 proxier.go:440] Adding new service "default/kube-dns:dns-tcp" at 10.0.0.10:53/TCP
kube-proxy[29691]: I1028 13:37:36.503055   29691 proxier.go:453] added serviceInfo(default/kube-dns:dns-tcp): (*iptables.serviceInfo)(0xc82049ba40)({
kube-proxy[29691]: clusterIP: (net.IP) (len=16 cap=16) 10.0.0.10,
kube-proxy[29691]: port: (int) 53,
kube-proxy[29691]: protocol: (api.Protocol) (len=3) "TCP",
kube-proxy[29691]: nodePort: (int) 0,
kube-proxy[29691]: loadBalancerStatus: (api.LoadBalancerStatus) {
kube-proxy[29691]: Ingress: ([]api.LoadBalancerIngress) {
kube-proxy[29691]: }
kube-proxy[29691]: },
kube-proxy[29691]: sessionAffinityType: (api.ServiceAffinity) (len=4) "None",
kube-proxy[29691]: stickyMaxAgeSeconds: (int) 180,
kube-proxy[29691]: externalIPs: ([]string) <nil>,
kube-proxy[29691]: loadBalancerSourceRanges: ([]string) <nil>
kube-proxy[29691]: })
kube-proxy[29691]: I1028 13:37:36.503062   29691 proxier.go:440] Adding new service "default/kubernetes:https" at 10.0.0.1:443/TCP
kube-proxy[29691]: I1028 13:37:36.503087   29691 proxier.go:453] added serviceInfo(default/kubernetes:https): (*iptables.serviceInfo)(0xc82049bae0)({
kube-proxy[29691]: clusterIP: (net.IP) (len=16 cap=16) 10.0.0.1,
kube-proxy[29691]: port: (int) 443,
kube-proxy[29691]: protocol: (api.Protocol) (len=3) "TCP",
kube-proxy[29691]: nodePort: (int) 0,
kube-proxy[29691]: loadBalancerStatus: (api.LoadBalancerStatus) {
kube-proxy[29691]: Ingress: ([]api.LoadBalancerIngress) {
kube-proxy[29691]: }
kube-proxy[29691]: },
kube-proxy[29691]: sessionAffinityType: (api.ServiceAffinity) (len=8) "ClientIP",
kube-proxy[29691]: stickyMaxAgeSeconds: (int) 180,
kube-proxy[29691]: externalIPs: ([]string) <nil>,
kube-proxy[29691]: loadBalancerSourceRanges: ([]string) <nil>
kube-proxy[29691]: })
kube-proxy[29691]: I1028 13:37:36.503123   29691 proxier.go:674] Not syncing iptables until Services and Endpoints have been received from master
kube-proxy[29691]: I1028 13:37:36.503128   29691 proxier.go:670] syncProxyRules took 18.524µs
kube-proxy[29691]: I1028 13:37:36.503135   29691 proxier.go:400] OnServiceUpdate took 201.564µs for 2 services
kube-proxy[29691]: I1028 13:37:36.503508   29691 config.go:99] Calling handler.OnEndpointsUpdate()
kube-proxy[29691]: I1028 13:37:36.503534   29691 proxier.go:516] Setting endpoints for "default/kubernetes:https" to [10.98.99.176:6443]
kube-proxy[29691]: I1028 13:37:36.503566   29691 proxier.go:677] Syncing iptables rules
kube-proxy[29691]: I1028 13:37:36.503571   29691 iptables.go:380] running iptables -N [KUBE-SERVICES -t filter]
kube-proxy[29691]: I1028 13:37:36.504519   29691 iptables.go:380] running iptables -N [KUBE-SERVICES -t nat]
kube-proxy[29691]: I1028 13:37:36.505365   29691 iptables.go:380] running iptables -C [OUTPUT -t filter -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:37:36.506177   29691 iptables.go:380] running iptables -C [OUTPUT -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:37:36.506976   29691 iptables.go:380] running iptables -C [PREROUTING -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:37:36.507794   29691 iptables.go:380] running iptables -N [KUBE-POSTROUTING -t nat]
kube-proxy[29691]: I1028 13:37:36.508626   29691 iptables.go:380] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes postrouting rules -j KUBE-POSTROUTING]
kube-proxy[29691]: I1028 13:37:36.509438   29691 iptables.go:299] running iptables-save [-t filter]
kube-proxy[29691]: I1028 13:37:36.510575   29691 iptables.go:299] running iptables-save [-t nat]
kube-proxy[29691]: I1028 13:37:36.511985   29691 proxier.go:1096] Restoring iptables rules: *filter
kube-proxy[29691]: :KUBE-SERVICES - [0:0]
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns has no endpoints" -m udp -p udp -d 10.0.0.10/32 --dport 53 -j REJECT
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns-tcp has no endpoints" -m tcp -p tcp -d 10.0.0.10/32 --dport 53 -j REJECT
kube-proxy[29691]: COMMIT
kube-proxy[29691]: *nat
kube-proxy[29691]: :KUBE-SERVICES - [0:0]
kube-proxy[29691]: :KUBE-NODEPORTS - [0:0]
kube-proxy[29691]: :KUBE-POSTROUTING - [0:0]
kube-proxy[29691]: :KUBE-MARK-MASQ - [0:0]
kube-proxy[29691]: :KUBE-SVC-D376NYSDDVFPF2KN - [0:0]
kube-proxy[29691]: :KUBE-SVC-N6R7PS4OMIK6NEO2 - [0:0]
kube-proxy[29691]: :KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
kube-proxy[29691]: :KUBE-SEP-SM34KKATJ2TS55C5 - [0:0]
kube-proxy[29691]: -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x00004000/0x00004000 -j MASQUERADE
kube-proxy[29691]: -A KUBE-MARK-MASQ -j MARK --set-xmark 0x00004000/0x00004000
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns cluster IP" -m udp -p udp -d 10.0.0.10/32 --dport 53 -j KUBE-SVC-D376NYSDDVFPF2KN
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns-tcp cluster IP" -m tcp -p tcp -d 10.0.0.10/32 --dport 53 -j KUBE-SVC-N6R7PS4OMIK6NEO2
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kubernetes:https cluster IP" -m tcp -p tcp -d 10.0.0.1/32 --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
kube-proxy[29691]: -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -m recent --name KUBE-SEP-SM34KKATJ2TS55C5 --rcheck --seconds 180 --reap -j KUBE-SEP-SM34KKATJ2TS55C5
kube-proxy[29691]: -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -j KUBE-SEP-SM34KKATJ2TS55C5
kube-proxy[29691]: -A KUBE-SEP-SM34KKATJ2TS55C5 -m comment --comment default/kubernetes:https -s 10.98.99.176/32 -j KUBE-MARK-MASQ
kube-proxy[29691]: -A KUBE-SEP-SM34KKATJ2TS55C5 -m comment --comment default/kubernetes:https -m recent --name KUBE-SEP-SM34KKATJ2TS55C5 --set -m tcp -p tcp -j DNAT --to-destination 10.98.99.176:6443
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
kube-proxy[29691]: COMMIT
kube-proxy[29691]: I1028 13:37:36.512060   29691 iptables.go:359] running iptables-restore [--noflush --counters /tmp/kube-temp-iptables-restore-332107562]
kube-proxy[29691]: I1028 13:37:36.517470   29691 conntrack.go:62] Setting nf_conntrack_tcp_timeout_established to 86400
kube-proxy[29691]: I1028 13:37:36.519918   29691 iptables.go:380] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes service traffic requiring SNAT -m mark --mark 0x4d415351 -j MASQUERADE]
kube-proxy[29691]: I1028 13:37:36.521108   29691 proxier.go:670] syncProxyRules took 17.541464ms
kube-proxy[29691]: I1028 13:37:36.521129   29691 proxier.go:478] OnEndpointsUpdate took 17.613002ms for 2 endpoints
kube-proxy[29691]: I1028 13:38:06.517827   29691 proxier.go:677] Syncing iptables rules
kube-proxy[29691]: I1028 13:38:06.517876   29691 iptables.go:380] running iptables -N [KUBE-SERVICES -t filter]
kube-proxy[29691]: I1028 13:38:06.519393   29691 iptables.go:380] running iptables -N [KUBE-SERVICES -t nat]
kube-proxy[29691]: I1028 13:38:06.520329   29691 iptables.go:380] running iptables -C [OUTPUT -t filter -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:38:06.521251   29691 iptables.go:380] running iptables -C [OUTPUT -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:38:06.522293   29691 iptables.go:380] running iptables -C [PREROUTING -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:38:06.523397   29691 iptables.go:380] running iptables -N [KUBE-POSTROUTING -t nat]
kube-proxy[29691]: I1028 13:38:06.524257   29691 iptables.go:380] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes postrouting rules -j KUBE-POSTROUTING]
kube-proxy[29691]: I1028 13:38:06.525331   29691 iptables.go:299] running iptables-save [-t filter]
kube-proxy[29691]: I1028 13:38:06.526562   29691 iptables.go:299] running iptables-save [-t nat]
kube-proxy[29691]: I1028 13:38:06.528202   29691 proxier.go:1096] Restoring iptables rules: *filter
kube-proxy[29691]: :KUBE-SERVICES - [0:0]
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns has no endpoints" -m udp -p udp -d 10.0.0.10/32 --dport 53 -j REJECT
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns-tcp has no endpoints" -m tcp -p tcp -d 10.0.0.10/32 --dport 53 -j REJECT
kube-proxy[29691]: COMMIT
kube-proxy[29691]: *nat
kube-proxy[29691]: :KUBE-SERVICES - [0:0]
kube-proxy[29691]: :KUBE-NODEPORTS - [0:0]
kube-proxy[29691]: :KUBE-POSTROUTING - [0:0]
kube-proxy[29691]: :KUBE-MARK-MASQ - [0:0]
kube-proxy[29691]: :KUBE-SVC-D376NYSDDVFPF2KN - [0:0]
kube-proxy[29691]: :KUBE-SVC-N6R7PS4OMIK6NEO2 - [0:0]
kube-proxy[29691]: :KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
kube-proxy[29691]: :KUBE-SEP-SM34KKATJ2TS55C5 - [0:0]
kube-proxy[29691]: -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x00004000/0x00004000 -j MASQUERADE
kube-proxy[29691]: -A KUBE-MARK-MASQ -j MARK --set-xmark 0x00004000/0x00004000
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns cluster IP" -m udp -p udp -d 10.0.0.10/32 --dport 53 -j KUBE-SVC-D376NYSDDVFPF2KN
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns-tcp cluster IP" -m tcp -p tcp -d 10.0.0.10/32 --dport 53 -j KUBE-SVC-N6R7PS4OMIK6NEO2
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kubernetes:https cluster IP" -m tcp -p tcp -d 10.0.0.1/32 --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
kube-proxy[29691]: -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -m recent --name KUBE-SEP-SM34KKATJ2TS55C5 --rcheck --seconds 180 --reap -j KUBE-SEP-SM34KKATJ2TS55C5
kube-proxy[29691]: -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -j KUBE-SEP-SM34KKATJ2TS55C5
kube-proxy[29691]: -A KUBE-SEP-SM34KKATJ2TS55C5 -m comment --comment default/kubernetes:https -s 10.98.99.176/32 -j KUBE-MARK-MASQ
kube-proxy[29691]: -A KUBE-SEP-SM34KKATJ2TS55C5 -m comment --comment default/kubernetes:https -m recent --name KUBE-SEP-SM34KKATJ2TS55C5 --set -m tcp -p tcp -j DNAT --to-destination 10.98.99.176:6443
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
kube-proxy[29691]: COMMIT
kube-proxy[29691]: I1028 13:38:06.528286   29691 iptables.go:359] running iptables-restore [--noflush --counters /tmp/kube-temp-iptables-restore-616375937]
kube-proxy[29691]: I1028 13:38:06.530293   29691 iptables.go:380] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes service traffic requiring SNAT -m mark --mark 0x4d415351 -j MASQUERADE]
kube-proxy[29691]: I1028 13:38:06.532051   29691 proxier.go:670] syncProxyRules took 14.232833ms

the kube-proxy receives the services but no rules are generate for them (maybe because the api-server does not provide the pod-ips)

here are my additional systemd startup-flags:

docker:      --bridge=cbr0 --iptables=false --ip-masq=false  
api-server:  --service-cluster-ip-range=10.0.0.0/24  
kubelet:     --configure-cbr0=true --cluster-dns=10.0.0.10  
kube-proxy:  --proxy-mode=iptables  

Answer 1

SOLVED:
as i investigated in this problem i queried the api to get more information, there i found this:
http://localhost:8080/api/v1/namespaces/default/endpoints :

...
notReadyAddresses: [
{
ip: "10.116.0.2",
targetRef: {
kind: "Pod",
namespace: "default",
name: "kube-dns-v10-kdhaf",
uid: "83d266e7-9ceb-11e6-bf42-5254009edb97",
resourceVersion: "535855"
}
}
],
...

the problem was that the dns-pod didn't start healthy since i activated ServiceAccount at admissioncontrol with some error-msg like ...missing serviceaccount-cert ...
after fixing that it runs like a charm