DOCKER学习_005:Flannel网络配置

一简介

Flannel是一种基于overlay网络的跨主机容器网络解决方案，也就是将TCP数据包封装在另一种网络包里面进行路由转发和通信，
Flannel是CoreOS开发,专门用于docker多机互联的一个工具,让集群中的不同节点主机创建的容器都具有全集群唯一的虚拟ip地址
Flannel使用go语言编写

二 Flannel实现原理

2.1原理说明

Flannel为每个host分配一个subnet，容器从这个subnet中分配IP，这些IP可以在host间路由，容器间无需使用nat和端口映射即可实现跨主机通信
每个subnet都是从一个更大的IP池中划分的，flannel会在每个主机上运行一个叫flanneld的agent，其职责就是从池子中分配subnet
Flannel使用etcd存放网络配置、已分配的subnet、host的IP等信息
Flannel数据包在主机间转发是由backend实现的，目前已经支持UDP、VxLAN、host-gw、AWS VPC和GCE路由等多种backend

2.2 数据转发流程

容器直接使用目标容器的ip访问，默认通过容器内部的eth0发送出去。
报文通过veth pair被发送到vethXXX。
vethXXX是直接连接到虚拟交换机docker0的，报文通过虚拟bridge docker0发送出去。
查找路由表，外部容器ip的报文都会转发到flannel0虚拟网卡，这是一个P2P的虚拟网卡，然后报文就被转发到监听在另一端的flanneld。
flanneld通过etcd维护了各个节点之间的路由表，把原来的报文UDP封装一层，通过配置的iface发送出去。
报文通过主机之间的网络找到目标主机。
报文继续往上，到传输层，交给监听在8285端口的flanneld程序处理。
数据被解包，然后发送给flannel0虚拟网卡。
查找路由表，发现对应容器的报文要交给docker0。
docker0找到连到自己的容器，把报文发送过去。

三 Flannel安装配置　

3.1 环境准备

节点名称	IP地址	安装软件
docker-server1	192.168.132.131	etcd、flannel、docker
docker-server2	192.168.132.132	flannel、docker

删掉节点的所有容器

[root@docker-server1 ~]# docker ps -aq|xargs docker rm

[root@docker-server2 ~]# docker ps -aq|xargs docker rm

3.2 安装etcd

etcd下载地址：https://github.com/coreos/etcd/releases

下载

[root@docker-server1 ~]# wget https://github.com/coreos/etcd/releases/download/v3.3.9/etcd-v3.3.9-linux-amd64.tar.gz

[root@docker-server2 ~]# wget https://github.com/coreos/etcd/releases/download/v3.3.9/etcd-v3.3.9-linux-amd64.tar.gz

在131上安装etcd和flannel

[root@docker-server1 ~]# tar -xf etcd-v3.3.9-linux-amd64.tar.gz

[root@docker-server1 ~]# cd etcd-v3.3.9-linux-amd64

[root@docker-server1 etcd-v3.3.9-linux-amd64]# ll

total 33992
drwxr-xr-x 11 joy joy     4096 Jul 24  2018 Documentation
-rwxr-xr-x  1 joy joy 18934016 Jul 24  2018 etcd
-rwxr-xr-x  1 joy joy 15809280 Jul 24  2018 etcdctl
-rw-r--r--  1 joy joy    38864 Jul 24  2018 README-etcdctl.md
-rw-r--r--  1 joy joy     7262 Jul 24  2018 README.md
-rw-r--r--  1 joy joy     7855 Jul 24  2018 READMEv2-etcdctl.md

[root@docker-server1 etcd-v3.3.9-linux-amd64]# cp etcd* /usr/bin/

启动命令：

[root@docker-server1 ~]# etcd -name etcd-131 -data-dir /var/lib/etcd --advertise-client-urls http://192.168.132.131:2379,http://127.0.0.1:2379 --listen-client-urls http://192.168.132.131:2379,http://127.0.0.1:2379

-name:etc取名

-data-dir:定义数据路径

2019-11-09 14:12:36.719599 I | etcdmain: Git SHA: fca8add78
2019-11-09 14:12:36.719603 I | etcdmain: Go Version: go1.10.3
2019-11-09 14:12:36.719606 I | etcdmain: Go OS/Arch: linux/amd64
2019-11-09 14:12:36.719613 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2019-11-09 14:12:36.720048 I | embed: listening for peers on http://localhost:2380
2019-11-09 14:12:36.720124 I | embed: listening for client requests on 127.0.0.1:2379
2019-11-09 14:12:36.720146 I | embed: listening for client requests on 192.168.132.131:2379
2019-11-09 14:12:36.722745 I | etcdserver: name = etcd-131
2019-11-09 14:12:36.722778 I | etcdserver: data dir = /var/lib/etcd
2019-11-09 14:12:36.722783 I | etcdserver: member dir = /var/lib/etcd/member
2019-11-09 14:12:36.722787 I | etcdserver: heartbeat = 100ms
2019-11-09 14:12:36.722791 I | etcdserver: election = 1000ms
2019-11-09 14:12:36.722794 I | etcdserver: snapshot count = 100000
2019-11-09 14:12:36.722811 I | etcdserver: advertise client URLs = http://127.0.0.1:2379,http://192.168.132.131:2379
2019-11-09 14:12:36.722816 I | etcdserver: initial advertise peer URLs = http://localhost:2380
2019-11-09 14:12:36.722823 I | etcdserver: initial cluster = etcd-131=http://localhost:2380
2019-11-09 14:12:36.725597 I | etcdserver: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32
2019-11-09 14:12:36.725645 I | raft: 8e9e05c52164694d became follower at term 0
2019-11-09 14:12:36.725658 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2019-11-09 14:12:36.725663 I | raft: 8e9e05c52164694d became follower at term 1
2019-11-09 14:12:36.731392 W | auth: simple token is not cryptographically signed
2019-11-09 14:12:36.732944 I | etcdserver: starting server... [version: 3.3.9, cluster version: to_be_decided]
2019-11-09 14:12:36.733497 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10)
2019-11-09 14:12:36.734281 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2019-11-09 14:12:37.635489 I | raft: 8e9e05c52164694d is starting a new election at term 1
2019-11-09 14:12:37.635568 I | raft: 8e9e05c52164694d became candidate at term 2
2019-11-09 14:12:37.635621 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2
2019-11-09 14:12:37.635656 I | raft: 8e9e05c52164694d became leader at term 2
2019-11-09 14:12:37.635676 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
2019-11-09 14:12:37.636216 I | etcdserver: setting up the initial cluster version to 3.3
2019-11-09 14:12:37.637689 N | etcdserver/membership: set the initial cluster version to 3.3
2019-11-09 14:12:37.637846 I | etcdserver/api: enabled capabilities for version 3.3
2019-11-09 14:12:37.637990 I | etcdserver: published {Name:etcd-131 ClientURLs:[http://127.0.0.1:2379 http://192.168.132.131:2379]} to cluster cdf818194e3a8c32
2019-11-09 14:12:37.638099 I | embed: ready to serve client requests
2019-11-09 14:12:37.638861 I | embed: ready to serve client requests
2019-11-09 14:12:37.639056 E | etcdmain: forgot to set Type=notify in systemd service file?
2019-11-09 14:12:37.640375 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2019-11-09 14:12:37.640424 N | embed: serving insecure client requests on 192.168.132.131:2379, this is strongly discouraged!

[root@docker-server1 ~]# ps -ef|grep etcd

root      85130  84636  2 14:12 pts/4    00:00:01 etcd -name etcd-131 -data-dir /var/lib/etcd --advertise-client-urls http://192.168.132.131:2379,http://127.0.0.1:2379 --listen-client-urls http://192.168.132.131:2379,http://127.0.0.1:2379

3.3 安装Flannel

flannel下载地址：https://github.com/coreos/flannel/releases

下载

[root@docker-server1 ~]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz

root@docker-server1 ~]# tar -xf flannel-v0.11.0-linux-amd64.tar.gz

[root@docker-server1 ~]# cp flanneld /usr/bin/
[root@docker-server1 ~]# cp mk-docker-opts.sh /usr/bin/

添加flannel网络配置信息到etcd：

[root@docker-server1 ~]# etcdctl set /coreos.com/network/config '{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}'

如果不是本机可以加参数：--endpoints http://IP:2379

{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}

Network：用于指定Flannel地址池

SubnetLen：用于指定分配给单个宿主机的docker0的ip段的子网掩码的长度

SubnetMin：用于指定最小能够分配的ip段

SudbnetMax：用于指定最大能够分配的ip段，在上面的示例中，表示每个宿主机可以分配一个24位掩码长度的子网，可以分配的子网从10.0.1.0/24到10.0.20.0/24，也就意味着在这个网段中，最多只能有20台宿主机

Backend：用于指定数据包以什么方式转发，默认为udp模式，host-gw模式性能最好，但不能跨宿主机网络

[root@docker-server1 ~]# etcdctl get /coreos.com/network/config

{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}

3.4 启动Flannel

[root@docker-server1 ~]# /usr/bin/flanneld --etcd-endpoints="http://192.168.132.131:2379" --iface=192.168.132.131 --etcd-prefix=/coreos.com/network &

[root@docker-server1 ~]# I1109 15:02:01.696813   86107 main.go:450] Searching for interface using 192.168.132.131
I1109 15:02:01.698311   86107 main.go:527] Using interface with name ens33 and address 192.168.132.131
I1109 15:02:01.698335   86107 main.go:544] Defaulting external address to interface address (192.168.132.131)
I1109 15:02:01.698556   86107 main.go:244] Created subnet manager: Etcd Local Manager with Previous Subnet: 10.0.4.0/24
I1109 15:02:01.698571   86107 main.go:247] Installing signal handlers
I1109 15:02:01.700808   86107 main.go:386] Found network config - Backend type: vxlan
I1109 15:02:01.700880   86107 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
I1109 15:02:01.703889   86107 local_manager.go:147] Found lease (10.0.4.0/24) for current IP (192.168.132.131), reusing
I1109 15:02:01.704954   86107 main.go:317] Wrote subnet file to /run/flannel/subnet.env
I1109 15:02:01.704967   86107 main.go:321] Running backend.
I1109 15:02:01.705791   86107 vxlan_network.go:60] watching for new subnet leases
I1109 15:02:01.709710   86107 main.go:429] Waiting for 22h59m59.994619649s to renew lease

[root@docker-server1 ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:91:dd:19 brd ff:ff:ff:ff:ff:ff
    inet 192.168.132.131/24 brd 192.168.132.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::bcf9:af19:a325:e2c7/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:3e:dd:55:81 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:3eff:fedd:5581/64 scope link 
       valid_lft forever preferred_lft forever
94: br-b1c2d9c1e522: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:cd:4a:25:4f brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global br-b1c2d9c1e522
       valid_lft forever preferred_lft forever
    inet6 fe80::42:cdff:fe4a:254f/64 scope link 
       valid_lft forever preferred_lft forever
95: br-ec4a8380b2d3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:c9:ba:23:ae brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-ec4a8380b2d3
       valid_lft forever preferred_lft forever
104: br-f42e46889a2a: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:b4:b6:35:7f brd ff:ff:ff:ff:ff:ff
    inet 172.22.16.1/24 brd 172.22.16.255 scope global br-f42e46889a2a
       valid_lft forever preferred_lft forever
    inet6 fe80::42:b4ff:feb6:357f/64 scope link 
       valid_lft forever preferred_lft forever
123: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 8e:b1:37:26:b0:59 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::8cb1:37ff:fe26:b059/64 scope link

[root@docker-server1 ~]# ps -ef |grep flan

root      86107  84636  0 15:02 pts/4    00:00:00 /usr/bin/flanneld --etcd-endpoints=http://192.168.132.131:2379 --iface=192.168.132.131 --etcd-prefix=/coreos.com/network

可以使用flannel提供的脚本将subnet.env转写成Docker启动参数，创建好的启动参数默认生成在/run/docker_opts.env文件中：

[root@docker-server1 ~]# mk-docker-opts.sh

[root@docker-server1 ~]# cat /run/docker_opts.env

DOCKER_OPT_BIP="--bip=10.0.4.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_OPTS=" --bip=10.0.4.1/24 --ip-masq=true --mtu=1450"

需要把这个文件加到docker的启动项

EnvironmentFile=/run/docker_opts.env
ExecStart=/usr/bin/dockerd  $DOCKER_OPTS  -H fd:// --containerd=/run/containerd/containerd.sock

启动docker

[root@docker-server1 ~]# systemctl daemon-reload

[root@docker-server1 ~]# systemctl restart docker

报错

[root@docker-server1 ~]# journalctl -xe -u docker

Nov 09 15:13:52 docker-server1 dockerd[86540]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following 
Nov 09 15:13:52 docker-server1 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE

删除/etc/docker/daemon.json里的bip配置

[root@docker-server1 ~]# systemctl restart docker

To force a start use "systemctl reset-failed docker.service" followed by "systemctl start docker.service" again.

[root@docker-server1 ~]# systemctl restart docker

[root@docker-server1 ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:91:dd:19 brd ff:ff:ff:ff:ff:ff
    inet 192.168.132.131/24 brd 192.168.132.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::bcf9:af19:a325:e2c7/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:3e:dd:55:81 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.1/24 brd 10.0.4.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:3eff:fedd:5581/64 scope link 
       valid_lft forever preferred_lft forever
94: br-b1c2d9c1e522: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:cd:4a:25:4f brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global br-b1c2d9c1e522
       valid_lft forever preferred_lft forever
    inet6 fe80::42:cdff:fe4a:254f/64 scope link 
       valid_lft forever preferred_lft forever
95: br-ec4a8380b2d3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:c9:ba:23:ae brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-ec4a8380b2d3
       valid_lft forever preferred_lft forever
104: br-f42e46889a2a: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:b4:b6:35:7f brd ff:ff:ff:ff:ff:ff
    inet 172.22.16.1/24 brd 172.22.16.255 scope global br-f42e46889a2a
       valid_lft forever preferred_lft forever
    inet6 fe80::42:b4ff:feb6:357f/64 scope link 
       valid_lft forever preferred_lft forever
123: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 8e:b1:37:26:b0:59 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::8cb1:37ff:fe26:b059/64 scope link 
       valid_lft forever preferred_lft forever

docker0的IP是10.0.4.1

可以看到flannel0网卡的地址和etcd存储的地址一样，这样flannel网络配置完成

Flannel启动过程解析：

从etcd中获取network的配置信息
划分subnet，并在etcd中进行注册
将子网信息记录到/run/flannel/subnet.env中
Flannel必须先于Docker启动

3.5 验证Flannel网络

查看etcd中的数据：

[root@docker-server1 ~]# etcdctl ls /coreos.com/network/subnets

/coreos.com/network/subnets/10.0.4.0-24

3.6 配置Docker

Docker安装完成以后，需要修改其启动参数以使其能够使用flannel进行IP分配，以及网络通讯

在Flannel运行之后，会生成一个环境变量文件，包含了当前主机要使用flannel通讯的相关参数，如下：

[root@docker-server1 ~]# cat /run/flannel/subnet.env

FLANNEL_NETWORK=10.0.0.0/16
FLANNEL_SUBNET=10.0.4.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

docker-server1相同操作配置flannel，不用安装etcd

[root@docker-server2 ~]# tar -xf flannel-v0.11.0-linux-amd64.tar.gz
[root@docker-server2 ~]# mv flanneld /usr/bin/

[root@docker-server2 ~]# mv mk-docker-opts.sh /usr/bin/

[root@docker-server2 ~]# /usr/bin/flanneld --etcd-endpoints="http://192.168.132.131:2379" --iface=192.168.132.132 --etcd-prefix=/coreos.com/network &

[root@docker-server2 ~]# mk-docker-opts.sh -c

[root@docker-server2 ~]# cat /run/docker_opts.env

修改docker启动文件，删除daemon.json的bip配置

[root@docker-server2 ~]# systemctl daemon-reload
[root@docker-server2 ~]# systemctl restart docker

[root@docker-server2 ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:63:fd:11 brd ff:ff:ff:ff:ff:ff
    inet 192.168.132.132/24 brd 192.168.132.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::6a92:62ba:1b33:c93d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:50:67:ff:90 brd ff:ff:ff:ff:ff:ff
    inet 10.0.18.1/24 brd 10.0.18.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:50ff:fe67:ff90/64 scope link 
       valid_lft forever preferred_lft forever
14: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 62:b9:76:44:bf:32 brd ff:ff:ff:ff:ff:ff
    inet 10.0.18.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::60b9:76ff:fe44:bf32/64 scope link 
       valid_lft forever preferred_lft forever

3.7 验证容器互通

[root@docker-server1 ~]# docker run -it busybox

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
124: eth0@if125: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue 
    link/ether 02:42:0a:00:04:02 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.2/24 brd 10.0.4.255 scope global eth0
       valid_lft forever preferred_lft forever

[root@docker-server2 ~]# docker run -it busybox

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
15: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue 
    link/ether 02:42:0a:00:12:02 brd ff:ff:ff:ff:ff:ff
    inet 10.0.18.2/24 brd 10.0.18.255 scope global eth0
       valid_lft forever preferred_lft forever

互ping

192.168.132.131容器
/ # ping 10.0.18.2
PING 10.0.18.2 (10.0.18.2): 56 data bytes
64 bytes from 10.0.18.2: seq=0 ttl=62 time=2.517 ms
64 bytes from 10.0.18.2: seq=1 ttl=62 time=1.058 ms
64 bytes from 10.0.18.2: seq=2 ttl=62 time=1.676 ms
192.168.132.132容器
PING 10.0.4.2 (10.0.4.2): 56 data bytes
64 bytes from 10.0.4.2: seq=0 ttl=62 time=2.606 ms
64 bytes from 10.0.4.2: seq=1 ttl=62 time=1.727 ms

两主机的容器可以互通

此时的网络数据包流向如图：

3.8 配置backend为host-gwhost-gw bakcend是flannel的另一个backend。与vxlan不同，host-gw不会封装数据包，而是在主机的路由表中创建到其他主机的subnet的路由条目，从而实现容器网络跨主机通信。需要说明的是，host-gw不能跨宿主机网络通信，或者说跨宿主机网络通信需要物理路由支持。
修改etcd如下：

[root@docker-server1 ~]# etcdctl --endpoints http://127.0.0.1:2379 set /coreos.com/network/config '{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "host-gw"}}'

{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "host-gw"}

重启flanneld与docker：

root@docker-server1 ~]# kill -9 86780

[root@docker-server1 ~]# /usr/bin/flanneld --etcd-endpoints="http://192.168.132.131:2379" --iface=192.168.132.131 --etcd-prefix=/coreos.com/network &

[root@docker-server1 ~]# systemctl restart docker

3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:3e:dd:55:81 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.1/24 brd 10.0.4.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:3eff:fedd:5581/64 scope link 
       valid_lft forever preferred_lft forever
123: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 8e:b1:37:26:b0:59 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::8cb1:37ff:fe26:b059/64 scope link 
       valid_lft forever preferred_lft forever

[root@docker-server2 ~]# kill -9 74050

[root@docker-server2 ~]# /usr/bin/flanneld --etcd-endpoints="http://192.168.132.131:2379" --iface=192.168.132.132 --etcd-prefix=/coreos.com/network &

[root@docker-server2 ~]# systemctl restart docker

[root@docker-server2 ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:63:fd:11 brd ff:ff:ff:ff:ff:ff
    inet 192.168.132.132/24 brd 192.168.132.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::6a92:62ba:1b33:c93d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:50:67:ff:90 brd ff:ff:ff:ff:ff:ff
    inet 10.0.18.1/24 brd 10.0.18.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:50ff:fe67:ff90/64 scope link 
       valid_lft forever preferred_lft forever
14: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 62:b9:76:44:bf:32 brd ff:ff:ff:ff:ff:ff
    inet 10.0.18.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::60b9:76ff:fe44:bf32/64 scope link 
       valid_lft forever preferred_lft forever

可以在宿主机上查看到路由条目：

[root@docker-server1 ~]# ip route

default via 192.168.132.2 dev ens33 proto static metric 100 
10.0.0.0/16 dev flannel.1 
10.0.4.0/24 dev docker0 proto kernel scope link src 10.0.4.1 
10.0.18.0/24 via 192.168.132.132 dev ens33 
172.17.0.0/16 dev br-b1c2d9c1e522 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev br-ec4a8380b2d3 proto kernel scope link src 172.18.0.1 
172.22.16.0/24 dev br-f42e46889a2a proto kernel scope link src 172.22.16.1 
192.168.132.0/24 dev ens33 proto kernel scope link src 192.168.132.131 metric 100

博主声明：本文的内容来源主要来自誉天教育晏威老师，由本人实验完成操作验证，需要的博友请联系誉天教育（http://www.yutianedu.com/），获得官方同意或者晏老师（https://www.cnblogs.com/breezey/）本人同意即可转载，谢谢！

来源：https://www.cnblogs.com/zyxnhr/p/11838414.html

标签

Docker

etcd

vxlan

host

DOCKER学习_005:Flannel网络配置

一 简介

二 Flannel实现原理

2.1原理说明

2.2 数据转发流程

三 Flannel安装配置

3.1 环境准备

3.2 安装etcd

3.3 安装Flannel

3.4 启动Flannel

3.5 验证Flannel网络

3.6 配置Docker

3.7 验证容器互通

一简介

三 Flannel安装配置　