问题
I have
- kubernetes v1.6.0 setup by kubeadm v1.6.1
- calico setup by offical yaml
- iptables v1.6.0
- nodes are provided by AliCloud
Problem:
The cni network is not working. Any deployment can only be visited from the node where it is running. I doubt it is related with route table conflict/missing, because I have another cluster on Vultr Cloud working fine, with the same setup steps.
Cluster Info:
root@iZ2ze8ctk2q17u029a8wcoZ:~# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system calico-etcd-66gf4 1/1 Running 0 16h 10.27.219.50 iz2ze8ctk2q17u029a8wcoz
kube-system calico-node-4wxsb 2/2 Running 0 16h 10.27.219.50 iz2ze8ctk2q17u029a8wcoz
kube-system calico-node-6n1g1 2/2 Running 0 16h 10.30.248.80 iz2zegw6nmd5t5qxy35lh0z
kube-system calico-policy-controller-2561685917-7bdd4 1/1 Running 0 16h 10.30.248.80 iz2zegw6nmd5t5qxy35lh0z
kube-system etcd-iz2ze8ctk2q17u029a8wcoz 1/1 Running 0 16h 10.27.219.50 iz2ze8ctk2q17u029a8wcoz
kube-system heapster-bx03l 1/1 Running 0 16h 192.168.31.150 iz2zegw6nmd5t5qxy35lh0z
kube-system kube-apiserver-iz2ze8ctk2q17u029a8wcoz 1/1 Running 0 16h 10.27.219.50 iz2ze8ctk2q17u029a8wcoz
kube-system kube-controller-manager-iz2ze8ctk2q17u029a8wcoz 1/1 Running 0 16h 10.27.219.50 iz2ze8ctk2q17u029a8wcoz
kube-system kube-dns-3913472980-kgzln 3/3 Running 0 16h 192.168.31.149 iz2zegw6nmd5t5qxy35lh0z
kube-system kube-proxy-ck83t 1/1 Running 0 16h 10.30.248.80 iz2zegw6nmd5t5qxy35lh0z
kube-system kube-proxy-lssdn 1/1 Running 0 16h 10.27.219.50 iz2ze8ctk2q17u029a8wcoz
kube-system kube-scheduler-iz2ze8ctk2q17u029a8wcoz 1/1 Running 0 16h 10.27.219.50 iz2ze8ctk2q17u029a8wcoz
I checked each pod's log, cannot find anything wrong.
Master Info: internal ip: 10.27.219.50
root@iZ2ze8ctk2q17u029a8wcoZ:~# ifconfig
docker0 Link encap:Ethernet HWaddr 02:42:56:84:35:19
inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr 00:16:3e:30:51:ae
inet addr:10.27.219.50 Bcast:10.27.219.255 Mask:255.255.252.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4400927 errors:0 dropped:0 overruns:0 frame:0
TX packets:3906530 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:564808928 (564.8 MB) TX bytes:792611382 (792.6 MB)
eth1 Link encap:Ethernet HWaddr 00:16:3e:32:07:f8
inet addr:59.110.32.199 Bcast:59.110.35.255 Mask:255.255.252.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1148756 errors:0 dropped:0 overruns:0 frame:0
TX packets:688177 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1570341044 (1.5 GB) TX bytes:58104611 (58.1 MB)
tunl0 Link encap:IPIP Tunnel HWaddr
inet addr:192.168.201.0 Mask:255.255.255.255
UP RUNNING NOARP MTU:1440 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
root@iZ2ze8ctk2q17u029a8wcoZ:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 59.110.35.247 0.0.0.0 UG 0 0 0 eth1
10.27.216.0 0.0.0.0 255.255.252.0 U 0 0 0 eth0
10.30.0.0 10.27.219.247 255.255.0.0 UG 0 0 0 eth0
10.32.0.0 0.0.0.0 255.240.0.0 U 0 0 0 weave
59.110.32.0 0.0.0.0 255.255.252.0 U 0 0 0 eth1
100.64.0.0 10.27.219.247 255.192.0.0 UG 0 0 0 eth0
172.16.0.0 10.27.219.247 255.240.0.0 UG 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
192.168.201.0 0.0.0.0 255.255.255.192 U 0 0 0 *
root@iZ2ze8ctk2q17u029a8wcoZ:~# ip route list
default via 59.110.35.247 dev eth1
10.27.216.0/22 dev eth0 proto kernel scope link src 10.27.219.50
10.30.0.0/16 via 10.27.219.247 dev eth0
10.32.0.0/12 dev weave proto kernel scope link src 10.32.0.1
59.110.32.0/22 dev eth1 proto kernel scope link src 59.110.32.199
100.64.0.0/10 via 10.27.219.247 dev eth0
172.16.0.0/12 via 10.27.219.247 dev eth0
172.17.0.0/24 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
blackhole 192.168.201.0/26 proto bird
// NOTE: 10.30.0.0/16 via 10.27.219.247 dev eth0
// this rule is important, the worker node's ip is 10.30.xx.xx. If I delete this rule, I cannot ping worker node.
// this rule is 10.0.0.0/8 via 10.27.219.247 dev eth0 by default, I changed it to the above.
root@iZ2ze8ctk2q17u029a8wcoZ:~# iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 3 packets, 180 bytes)
pkts bytes target prot opt in out source destination
20976 1250K cali-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:6gwbT8clXdHdC1b1 */
21016 1252K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
20034 1193K DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 3 packets, 180 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 4 packets, 240 bytes)
pkts bytes target prot opt in out source destination
109K 6580K cali-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:tVnHkvAo15HuiPy0 */
111K 6738K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
1263 75780 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 4 packets, 240 bytes)
pkts bytes target prot opt in out source destination
86584 5235K cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:O3lYWMrLQYEMJtB5 */
0 0 MASQUERADE all -- * !docker0 172.17.0.0/24 0.0.0.0/0
3982K 239M KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
28130 1704K WEAVE all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-MARK-DROP (0 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000
Chain KUBE-MARK-MASQ (5 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000
Chain KUBE-NODEPORTS (1 references)
pkts bytes target prot opt in out source destination
Chain KUBE-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
Chain KUBE-SEP-2VS52M6CEWASZVOP (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.31.149 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.31.149:53
Chain KUBE-SEP-3XQHSFTDAPNNNDX3 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.31.150 0.0.0.0/0 /* kube-system/heapster: */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/heapster: */ tcp to:192.168.31.150:8082
Chain KUBE-SEP-CH7KJM5XKO5WGA6D (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.27.219.50 0.0.0.0/0 /* default/kubernetes:https */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ recent: SET name: KUBE-SEP-CH7KJM5XKO5WGA6D side: source mask: 255.255.255.255 tcp to:10.27.219.50:6443
Chain KUBE-SEP-X3WTOMIYJNS7APAN (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.31.149 0.0.0.0/0 /* kube-system/kube-dns:dns */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:192.168.31.149:53
Chain KUBE-SEP-YDCHDMTZNPMRRKCX (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.27.219.50 0.0.0.0/0 /* kube-system/calico-etcd: */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/calico-etcd: */ tcp to:10.27.219.50:6666
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
0 0 KUBE-SVC-NTYB37XIWATNM25Y tcp -- * * 0.0.0.0/0 10.96.232.136 /* kube-system/calico-etcd: cluster IP */ tcp dpt:6666
0 0 KUBE-SVC-BJM46V3U5RZHCFRZ tcp -- * * 0.0.0.0/0 10.96.181.180 /* kube-system/heapster: cluster IP */ tcp dpt:80
7 420 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Chain KUBE-SVC-BJM46V3U5RZHCFRZ (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-3XQHSFTDAPNNNDX3 all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/heapster: */
Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-2VS52M6CEWASZVOP all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-CH7KJM5XKO5WGA6D all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-CH7KJM5XKO5WGA6D side: source mask: 255.255.255.255
0 0 KUBE-SEP-CH7KJM5XKO5WGA6D all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */
Chain KUBE-SVC-NTYB37XIWATNM25Y (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-YDCHDMTZNPMRRKCX all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/calico-etcd: */
Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-X3WTOMIYJNS7APAN all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */
Chain WEAVE (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 10.32.0.0/12 224.0.0.0/4
1 93 MASQUERADE all -- * * !10.32.0.0/12 10.32.0.0/12
0 0 MASQUERADE all -- * * 10.32.0.0/12 !10.32.0.0/12
Chain cali-OUTPUT (1 references)
pkts bytes target prot opt in out source destination
109K 6580K cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:GBTAv2p5CwevEyJm */
Chain cali-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
109K 6571K cali-fip-snat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Z-c7XtVd2Bq7s_hA */
109K 6571K cali-nat-outgoing all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:nYKhEzDlr11Jccal */
0 0 MASQUERADE all -- * tunl0 0.0.0.0/0 0.0.0.0/0 /* cali:JHlpT-eSqR1TvyYm */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL
Chain cali-PREROUTING (1 references)
pkts bytes target prot opt in out source destination
20976 1250K cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:r6XmIziWUJsdOK6Z */
Chain cali-fip-dnat (2 references)
pkts bytes target prot opt in out source destination
Chain cali-fip-snat (1 references)
pkts bytes target prot opt in out source destination
Chain cali-nat-outgoing (1 references)
pkts bytes target prot opt in out source destination
4 376 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Wd76s91357Uv7N3v */ match-set cali4-masq-ipam-pools src ! match-set cali4-all-ipam-pools dst
Worker Node Info: internal ip: 10.30.248.80
ifconfig
docker0 Link encap:Ethernet HWaddr 02:42:58:2b:b5:39
inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr 00:16:3e:2e:3d:fd
inet addr:10.30.248.80 Bcast:10.30.251.255 Mask:255.255.252.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3856596 errors:0 dropped:0 overruns:0 frame:0
TX packets:4253613 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:827402268 (827.4 MB) TX bytes:510838231 (510.8 MB)
eth1 Link encap:Ethernet HWaddr 00:16:3e:2c:db:d1
inet addr:47.93.161.177 Bcast:47.93.163.255 Mask:255.255.252.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:890451 errors:0 dropped:0 overruns:0 frame:0
TX packets:825607 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1695352720 (1.6 GB) TX bytes:62341312 (62.3 MB)
tunl0 Link encap:IPIP Tunnel HWaddr
inet addr:192.168.31.128 Mask:255.255.255.255
UP RUNNING NOARP MTU:1440 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
root@iZ2zegw6nmd5t5qxy35lh0Z:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 47.93.163.247 0.0.0.0 UG 0 0 0 eth1
10.0.0.0 10.30.251.247 255.0.0.0 UG 0 0 0 eth0
10.30.248.0 0.0.0.0 255.255.252.0 U 0 0 0 eth0
47.93.160.0 0.0.0.0 255.255.252.0 U 0 0 0 eth1
100.64.0.0 10.30.251.247 255.192.0.0 UG 0 0 0 eth0
172.16.0.0 10.30.251.247 255.240.0.0 UG 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
192.168.31.128 0.0.0.0 255.255.255.192 U 0 0 0 *
192.168.31.149 0.0.0.0 255.255.255.255 UH 0 0 0 cali3567b3362cc
192.168.31.150 0.0.0.0 255.255.255.255 UH 0 0 0 cali9d04015b0e7
root@iZ2zegw6nmd5t5qxy35lh0Z:~# ip route list
default via 47.93.163.247 dev eth1
10.0.0.0/8 via 10.30.251.247 dev eth0
10.30.248.0/22 dev eth0 proto kernel scope link src 10.30.248.80
47.93.160.0/22 dev eth1 proto kernel scope link src 47.93.161.177
100.64.0.0/10 via 10.30.251.247 dev eth0
172.16.0.0/12 via 10.30.251.247 dev eth0
172.17.0.0/24 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
blackhole 192.168.31.128/26 proto bird
192.168.31.149 dev cali3567b3362cc scope link
192.168.31.150 dev cali9d04015b0e7 scope link
// NOTE: 10.0.0.0/8 via 10.30.251.247 dev eth0
// I didn't change this one. So it is default now.
root@iZ2zegw6nmd5t5qxy35lh0Z:~# iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
3524 263K cali-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:6gwbT8clXdHdC1b1 */
3527 263K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
1031 53882 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 4 packets, 240 bytes)
pkts bytes target prot opt in out source destination
84174 5099K cali-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:tVnHkvAo15HuiPy0 */
85201 5163K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 7 packets, 420 bytes)
pkts bytes target prot opt in out source destination
76279 4644K cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:O3lYWMrLQYEMJtB5 */
0 0 MASQUERADE all -- * !docker0 172.17.0.0/24 0.0.0.0/0
87179 5342K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
43815 2646K WEAVE all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-MARK-DROP (0 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000
Chain KUBE-MARK-MASQ (5 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000
Chain KUBE-NODEPORTS (1 references)
pkts bytes target prot opt in out source destination
Chain KUBE-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
Chain KUBE-SEP-2VS52M6CEWASZVOP (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.31.149 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.31.149:53
Chain KUBE-SEP-3XQHSFTDAPNNNDX3 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.31.150 0.0.0.0/0 /* kube-system/heapster: */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/heapster: */ tcp to:192.168.31.150:8082
Chain KUBE-SEP-CH7KJM5XKO5WGA6D (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.27.219.50 0.0.0.0/0 /* default/kubernetes:https */
3 180 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ recent: SET name: KUBE-SEP-CH7KJM5XKO5WGA6D side: source mask: 255.255.255.255 tcp to:10.27.219.50:6443
Chain KUBE-SEP-X3WTOMIYJNS7APAN (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.31.149 0.0.0.0/0 /* kube-system/kube-dns:dns */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:192.168.31.149:53
Chain KUBE-SEP-YDCHDMTZNPMRRKCX (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.27.219.50 0.0.0.0/0 /* kube-system/calico-etcd: */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/calico-etcd: */ tcp to:10.27.219.50:6666
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
3 180 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
0 0 KUBE-SVC-NTYB37XIWATNM25Y tcp -- * * 0.0.0.0/0 10.96.232.136 /* kube-system/calico-etcd: cluster IP */ tcp dpt:6666
0 0 KUBE-SVC-BJM46V3U5RZHCFRZ tcp -- * * 0.0.0.0/0 10.96.181.180 /* kube-system/heapster: cluster IP */ tcp dpt:80
0 0 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Chain KUBE-SVC-BJM46V3U5RZHCFRZ (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-3XQHSFTDAPNNNDX3 all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/heapster: */
Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-2VS52M6CEWASZVOP all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
pkts bytes target prot opt in out source destination
3 180 KUBE-SEP-CH7KJM5XKO5WGA6D all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-CH7KJM5XKO5WGA6D side: source mask: 255.255.255.255
0 0 KUBE-SEP-CH7KJM5XKO5WGA6D all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */
Chain KUBE-SVC-NTYB37XIWATNM25Y (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-YDCHDMTZNPMRRKCX all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/calico-etcd: */
Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-X3WTOMIYJNS7APAN all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */
Chain WEAVE (1 references)
pkts bytes target prot opt in out source destination
Chain cali-OUTPUT (1 references)
pkts bytes target prot opt in out source destination
84174 5099K cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:GBTAv2p5CwevEyJm */
Chain cali-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
86501 5298K cali-fip-snat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Z-c7XtVd2Bq7s_hA */
86501 5298K cali-nat-outgoing all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:nYKhEzDlr11Jccal */
0 0 MASQUERADE all -- * tunl0 0.0.0.0/0 0.0.0.0/0 /* cali:JHlpT-eSqR1TvyYm */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL
Chain cali-PREROUTING (1 references)
pkts bytes target prot opt in out source destination
3524 263K cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:r6XmIziWUJsdOK6Z */
Chain cali-fip-dnat (2 references)
pkts bytes target prot opt in out source destination
Chain cali-fip-snat (1 references)
pkts bytes target prot opt in out source destination
Chain cali-nat-outgoing (1 references)
pkts bytes target prot opt in out source destination
29 1726 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Wd76s91357Uv7N3v */ match-set cali4-masq-ipam-pools src ! match-set cali4-all-ipam-pools dst
回答1:
I'm not sure what the problem is but here are a couple things to consider:
- I am not familiar with AliCloud but sometimes there are special consideration for some cloud providers. For example with GCE the IP-in-IP must be explicitly allowed, http://docs.projectcalico.org/v2.1/getting-started/kubernetes/installation/gce.
- I see the weave interface on your master so I'm wondering if weave could have left something around that is causing a problem.
- Also as was suggested in your issue https://github.com/projectcalico/cni-plugin/issues/314 you should check
calicoctl node statuson the nodes to see if BGP is working as expected.
回答2:
Problem is found by calicoctl node status. The calico/node use a public ip to communicate with each other. But nodes in AliCloud are behind a firewall. So they cannot do that via public ip address.
As gunjan5 suggested, I used this env var IP_AUTODETECTION_METHOD to specify the internal interface. Problem solved.
来源：https://stackoverflow.com/questions/43485770/route-not-working-in-kubernetes-with-calico