首页 > 系统相关> > k8s实践4:kube-proxy问题iptables转发规则不显示

k8s实践4:kube-proxy问题iptables转发规则不显示

2019-03-12 16:56:30 作者：互联网

1.
今天想看看kube-proxy的iptables转发规则,执行命令iptables-save,见下:

[root@k8s-master1 test]# iptables-save |grep "^-A KUBE"
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 172.30.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 172.30.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-SERVICES ! -s 172.30.0.0/16 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT

发现,根本看不到详细的svc的和pod的iptables转发规则.
为什么看不到呢？以前kubeadm部署是可以看到的.

2.
查资料做对比.
发现kubeadm部署使用kube-proxy的ipatbles模式
现在手动部署的集群使用的是kube-proxy的ipvs模式

可以通过检查配置文件,检索模式?

[root@k8s-master2 ~]# cat /etc/kubernetes/kube-proxy.config.yaml

apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.32.129
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 172.30.0.0/16
healthzBindAddress: 192.168.32.129:10256
hostnameOverride: k8s-master2
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.32.129:10249
mode: "ipvs"

把ipvs模式改成iptables,重启kube-proxy.
执行命令iptables-save命令,见下:

[root@k8s-master2 ~]# iptables-save |grep "^-A KUBE-SVC"
-A KUBE-SVC-ERIFXISQEP7F7OF4 -j KUBE-SEP-GU5WDSRFVPTYJ5QU
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-TZ3VXNY2EEVCTOTN
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-BU2YJ53RTO4VMWUK
-A KUBE-SVC-NPX46M4PTMTKRN6Y -j KUBE-SEP-NOTEWJDBBN5H3PPR
-A KUBE-SVC-R2VK7O5AFVLRAXSH -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-55H4YX2333AS44RT
-A KUBE-SVC-R2VK7O5AFVLRAXSH -j KUBE-SEP-YZEC6Y7BVIHLFR3L
-A KUBE-SVC-RL3JAE4GN7VOGDGP -m statistic --mode random --probability 0.20000000019 -j KUBE-SEP-VB7GMOVJXYUHR5XB
-A KUBE-SVC-RL3JAE4GN7VOGDGP -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-SOERVJR7HCE5UQCC
-A KUBE-SVC-RL3JAE4GN7VOGDGP -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-2EMOF3UUDCCOTQCO
-A KUBE-SVC-RL3JAE4GN7VOGDGP -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ZMLJTAH443KZVOBZ
-A KUBE-SVC-RL3JAE4GN7VOGDGP -j KUBE-SEP-OTUMA5HXXG654BGO
-A KUBE-SVC-TCOU7JCQXEZGVUNU -j KUBE-SEP-4H6QT2QBUKHBI7U2
-A KUBE-SVC-VPEW22VBKQ5JUV7N -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-6ZCL2K4RTLLEWKG3
-A KUBE-SVC-VPEW22VBKQ5JUV7N -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-HBKM2LVSQ4YLR7GU
-A KUBE-SVC-VPEW22VBKQ5JUV7N -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-NGAS2WWEJXLUMON5
-A KUBE-SVC-VPEW22VBKQ5JUV7N -j KUBE-SEP-ZEKL4SKLF2DTYX5K
[root@k8s-master2 ~]#

[root@k8s-master2 ~]# iptables-save |grep "^-A KUBE-SEP"
-A KUBE-SEP-2EMOF3UUDCCOTQCO -s 172.30.78.2/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-2EMOF3UUDCCOTQCO -p tcp -m tcp -j DNAT --to-destination 172.30.78.2:80
-A KUBE-SEP-4H6QT2QBUKHBI7U2 -s 172.30.60.4/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-4H6QT2QBUKHBI7U2 -p udp -m udp -j DNAT --to-destination 172.30.60.4:53
-A KUBE-SEP-55H4YX2333AS44RT -s 172.30.60.5/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-55H4YX2333AS44RT -p tcp -m tcp -j DNAT --to-destination 172.30.60.5:80
-A KUBE-SEP-6ZCL2K4RTLLEWKG3 -s 172.30.60.2/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-6ZCL2K4RTLLEWKG3 -p tcp -m tcp -j DNAT --to-destination 172.30.60.2:80
-A KUBE-SEP-BU2YJ53RTO4VMWUK -s 192.168.32.129/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-BU2YJ53RTO4VMWUK -p tcp -m tcp -j DNAT --to-destination 192.168.32.129:6443
-A KUBE-SEP-GU5WDSRFVPTYJ5QU -s 172.30.60.4/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-GU5WDSRFVPTYJ5QU -p tcp -m tcp -j DNAT --to-destination 172.30.60.4:53
-A KUBE-SEP-HBKM2LVSQ4YLR7GU -s 172.30.7.2/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-HBKM2LVSQ4YLR7GU -p tcp -m tcp -j DNAT --to-destination 172.30.7.2:80
-A KUBE-SEP-NGAS2WWEJXLUMON5 -s 172.30.78.3/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-NGAS2WWEJXLUMON5 -p tcp -m tcp -j DNAT --to-destination 172.30.78.3:80
-A KUBE-SEP-NOTEWJDBBN5H3PPR -s 192.168.32.130/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-NOTEWJDBBN5H3PPR -p tcp -m tcp -j DNAT --to-destination 192.168.32.130:6443
-A KUBE-SEP-OTUMA5HXXG654BGO -s 172.30.80.3/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-OTUMA5HXXG654BGO -p tcp -m tcp -j DNAT --to-destination 172.30.80.3:80
-A KUBE-SEP-SOERVJR7HCE5UQCC -s 172.30.7.3/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-SOERVJR7HCE5UQCC -p tcp -m tcp -j DNAT --to-destination 172.30.7.3:80
-A KUBE-SEP-TZ3VXNY2EEVCTOTN -s 192.168.32.128/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-TZ3VXNY2EEVCTOTN -p tcp -m tcp -j DNAT --to-destination 192.168.32.128:6443
-A KUBE-SEP-VB7GMOVJXYUHR5XB -s 172.30.60.3/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-VB7GMOVJXYUHR5XB -p tcp -m tcp -j DNAT --to-destination 172.30.60.3:80
-A KUBE-SEP-YZEC6Y7BVIHLFR3L -s 172.30.78.4/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-YZEC6Y7BVIHLFR3L -p tcp -m tcp -j DNAT --to-destination 172.30.78.4:80
-A KUBE-SEP-ZEKL4SKLF2DTYX5K -s 172.30.80.4/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-ZEKL4SKLF2DTYX5K -p tcp -m tcp -j DNAT --to-destination 172.30.80.4:80
-A KUBE-SEP-ZMLJTAH443KZVOBZ -s 172.30.80.2/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-ZMLJTAH443KZVOBZ -p tcp -m tcp -j DNAT --to-destination 172.30.80.2:80
[root@k8s-master2 ~]#

可以看到改回iptables模式之后,所有的转发规则全部显示出来了.

3.
ipvs模式比iptables模式强大,测试完后,记得改回ipvs模式.

标签：iptables,KUBE,SEP,tcp,MARK,proxy,MASQ,kube,172.30
来源： https://blog.51cto.com/goome/2361857