一、Docker网络命名空间
1、利用busybox启动两个容器
- 启动test1容器
[root@localhost ~]# docker run -d --name test1 busybox /bin/sh -c "while true; do sleep 3600; done " #启动test1容器
- 启动test2容器
[root@localhost ~]# docker run -d --name test2 busybox /bin/sh -c "while true; do sleep 3600; done "
2、进入容器中查看网络
- 进入test1容器查看网络
[root@localhost ~]# docker exec -it 90964ccfc53d /bin/sh / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0 valid_lft forever preferred_lft forever / #
- 进入test2容器查看网络
[root@localhost ~]# docker exec -it 08d4fa600414 /bin/sh / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0 valid_lft forever preferred_lft forever
可以看到两个容器产生的ip不一样,也就是说每一个容器有自己的网络,那么它们之间是否可以通信呢?
我们可以在test2容器中尝试ping第一个容器:
[root@localhost ~]# docker exec -it 90964ccfc53d /bin/sh / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0 valid_lft forever preferred_lft forever / # ping 172.17.0.2 PING 172.17.0.2 (172.17.0.2): 56 data bytes 64 bytes from 172.17.0.2: seq=0 ttl=64 time=0.194 ms 64 bytes from 172.17.0.2: seq=1 ttl=64 time=0.068 ms 64 bytes from 172.17.0.2: seq=2 ttl=64 time=0.088 ms 64 bytes from 172.17.0.2: seq=3 ttl=64 time=0.070 ms 64 bytes from 172.17.0.2: seq=4 ttl=64 time=0.100 ms
可以看到是可以进行通信的,所以每一个容器都有自己的网络命名空间,并且容器之间是可以进行通信的。
二、Docker网络
我们可以先看看Docker中的网络情况:
[root@localhost ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 4657ce390049 bridge bridge local 2582de0db573 host host local d81e2ab77dcf none null local
可以看到有bridge、host、以及none三种情况。
(一) bridge
1、bridge网络详情
通过docker network inspect + id查看具体网络情况:
[root@localhost ~]# docker network inspect 4657ce390049 [ { "Name": "bridge", "Id": "4657ce390049c246dfb67dad34ad8e456958b2875d71fb2214269aa71642eda2", "Created": "2020-01-31T10:04:59.74388944+08:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.17.0.0/16", "Gateway": "172.17.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "08d4fa6004147b171820d7f81903e7af7ccff38498082365383ca4855e9d6482": { "Name": "test2", "EndpointID": "c2d18508f072d899f55302e2277976c43895163b0b58b95fbabc08041d9dd55a", "MacAddress": "02:42:ac:11:00:02", "IPv4Address": "172.17.0.2/16", "IPv6Address": "" }, "90964ccfc53d5a8c4a35a0e53f2f022205df39294daacbd6175703a704688cd4": { "Name": "test3", "EndpointID": "7f08319a7959f34be312ea03d21353edeb21b12ed4dac619b83b4c9c86065625", "MacAddress": "02:42:ac:11:00:03", "IPv4Address": "172.17.0.3/16", "IPv6Address": "" } }, "Options": { "com.docker.network.bridge.default_bridge": "true", "com.docker.network.bridge.enable_icc": "true", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0", "com.docker.network.bridge.name": "docker0", "com.docker.network.driver.mtu": "1500" }, "Labels": {} } ]
可以看到这是Bridge网络的详情,其中container中有test2这个容器使用的就是这个网络。
2、容器之间的通信
容器之间是如何进行通信的呢?
- 一个容器的网络
我们先看看主机上的网络:
[root@localhost ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:f4:16:7e brd ff:ff:ff:ff:ff:ff inet 192.168.0.109/24 brd 192.168.0.255 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::84a4:73f5:46d2:79d3/64 scope link valid_lft forever preferred_lft forever 3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000 link/ether 52:54:00:de:c3:4c brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000 link/ether 52:54:00:de:c3:4c brd ff:ff:ff:ff:ff:ff 5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether 02:42:a2:8a:59:a5 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:a2ff:fe8a:59a5/64 scope link valid_lft forever preferred_lft forever 7: veth309f7b4@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP link/ether be:d5:f3:b1:0b:0b brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::bcd5:f3ff:feb1:b0b/64 scope link valid_lft forever preferred_lft forever
可以看到上面的网络有很多,但是注意docker0这个网络,目前运行的容器有一个:
[root@localhost ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 08d4fa600414 busybox "/bin/sh -c 'while t…" About an hour ago Up About an hour test2
那么上面第7个网络veth309f7b4其实连接的就是docker0这个网络,我们可以通过以下命令查看:
[root@localhost ~]# brctl show bridge name bridge id STP enabled interfaces docker0 8000.0242a28a59a5 no veth309f7b4
可以看到interfaces中值和第7个是一样的。
- 两个容器的网路
我们再运行一个容器:
[root@localhost ~]# docker run -d --name test3 busybox /bin/sh -c "while true; do sleep 3600; done " 74859a31d0bf3d0d8a2664a3de7a95a54576350b79bb764b9f5085d66d258b1a [root@localhost ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 74859a31d0bf busybox "/bin/sh -c 'while t…" 10 seconds ago Up 4 seconds test3 08d4fa600414 busybox "/bin/sh -c 'while t…" About an hour ago Up About an hour test2
目前有两个容器,再看看网络情况,首先主机网络情况:
[root@localhost ~]# ip a ... ... 5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether 02:42:a2:8a:59:a5 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:a2ff:fe8a:59a5/64 scope link valid_lft forever preferred_lft forever 7: veth309f7b4@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP link/ether be:d5:f3:b1:0b:0b brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::bcd5:f3ff:feb1:b0b/64 scope link valid_lft forever preferred_lft forever 15: veth48166c0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP link/ether 5a:f2:df:c2:7f:80 brd ff:ff:ff:ff:ff:ff link-netnsid 1 inet6 fe80::58f2:dfff:fec2:7f80/64 scope link valid_lft forever preferred_lft forever
我们观察多了一个15的网络,那么它连接的是谁呢?
[root@localhost ~]# brctl show bridge name bridge id STP enabled interfaces docker0 8000.0242a28a59a5 no veth309f7b4 veth48166c0
可以看到它连接的还是docker0这个Bridge网络。另外可以看到test3容器的网络情况:
[root@localhost ~]# docker exec -it 74859a31d0bf ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0 valid_lft forever preferred_lft forever
- 总结
可以看到每一个容器与docker0之间的相连都是通过一对接口进行连接,这样通过docker0间接的达到容器之间的通信。
- 容器之间的link
上面我们可以看出来,容器之间ping通是需要通过ip地址进行通信的,那么link的好处就是在启动容器时,通过link参数直接指定容器的名称,这样就可以直接通过容器名称通信而不需要ip地址:
创建容器test1:
[root@localhost ~]# docker run -d --name test1 busybox /bin/sh -c "while true; do sleep 3600; done " b5ab3edf65f7687cc95cd23d01c5dd04bbc15c291f231a38a684ade889908ecd
创建容器test2,并且通过link参数指定test1:
[root@localhost ~]# docker run -d --name test2 --link test1 busybox /bin/sh -c "while true; do sleep 3600; done " cd8e333b7c96388318c9a5d9436997fea8a722343ca745dcbe60d5e256cf4433
这样,我们可以进入到test2中,然后去尝试通过名称连接test1:
[root@localhost ~]# docker exec -it test2 /bin/sh / # ping test1 PING test1 (172.17.0.2): 56 data bytes 64 bytes from 172.17.0.2: seq=0 ttl=64 time=0.235 ms 64 bytes from 172.17.0.2: seq=1 ttl=64 time=0.087 ms 64 bytes from 172.17.0.2: seq=2 ttl=64 time=0.090 ms
显然,这样是可行的,但是如果我们进入到test1,然后通过name去连接test2的话是会失败的,因为link是有指向性的。这个是默认的bridge所限制的,但是如果是自己搭建的网络bridge就不是这种情况了。
3、自创建bridge网络
在此之前,先再看一看主机上的网络情况:
[root@localhost ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 3198a081bd7f bridge bridge local 2582de0db573 host host local d81e2ab77dcf none null local
- 创建bridge网络
[root@localhost ~]# docker network create -d bridge my-bridge 2d6d1e198a6c2e3f98c1b6afcac19a4a7cd106d1da1029d41a73c00a0729fd5f [root@localhost ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 3198a081bd7f bridge bridge local 2582de0db573 host host local 2d6d1e198a6c my-bridge bridge local d81e2ab77dcf none null local
目前还没有容器来连接它:
[root@localhost ~]# brctl show bridge name bridge id STP enabled interfaces br-2d6d1e198a6c 8000.0242ae7aebe7 no docker0 8000.024287a39934 no vethbc08bb5 vethf24eba9
- 创建容器test3来连接my-bridge网络
[root@localhost ~]# docker run -d --name test3 --network my-bridge busybox /bin/sh -c "while true; do sleep 3600; done " e87b1468c7df57080177500135f6dd0e935894c447cfda47bae395affe5f58d0
此时再看,已经有一个接口了:
[root@localhost ~]# brctl show bridge name bridge id STP enabled interfaces br-2d6d1e198a6c 8000.0242ae7aebe7 no vethb722bba docker0 8000.024287a39934 no vethbc08bb5 vethf24eba9
- 让容器test2的网络由bridge变成my-bridge
[root@localhost ~]# docker network connect my-bridge test2 [root@localhost ~]# brctl show bridge name bridge id STP enabled interfaces br-2d6d1e198a6c 8000.0242ae7aebe7 no vethb722bba vethd2f00a0 docker0 8000.024287a39934 no vethbc08bb5 vethf24eba9
可以看到my-bridge上已经有两个接口了,此时我们进入到其中的一个容器然后通过name去通信。
- my-bridge上的容器互相通信
[root@localhost ~]# docker exec -it test2 /bin/sh #进入test2 / # ping test3 PING test3 (172.18.0.2): 56 data bytes 64 bytes from 172.18.0.2: seq=0 ttl=64 time=1.972 ms 64 bytes from 172.18.0.2: seq=1 ttl=64 time=0.096 ms 64 bytes from 172.18.0.2: seq=2 ttl=64 time=0.081 ms [root@localhost ~]# docker exec -it test3 /bin/sh #进入test3 / # ping test2 PING test2 (172.18.0.3): 56 data bytes 64 bytes from 172.18.0.3: seq=0 ttl=64 time=0.141 ms 64 bytes from 172.18.0.3: seq=1 ttl=64 time=0.076 ms 64 bytes from 172.18.0.3: seq=2 ttl=64 time=0.078 ms
- 总结
如果是自建的bridge网络,不需要通过link参数,而是本身自己在连接my-bridge已经ip和name进行了映射,我们通过名字就可以了,类似于DNS。
(二)host、none
1、host
- 创建host网络的容器
[root@localhost ~]# docker run -d --name test1 --network host busybox /bin/sh -c "while true; do sleep 3600; done " cdedc9c7525bdfad4642c95d1d0c2dfc1b018b4a28672e1547f2b55391c9dbdf
- 查看host网络
[root@localhost ~]# docker network inspect host [ { "Name": "host", "Id": "2582de0db573f47b1cf71cb691f5f2d72e0626b0f2574c5a861f582b981388a3", "Created": "2019-07-02T22:55:28.657713062+08:00", "Scope": "local", "Driver": "host", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "cdedc9c7525bdfad4642c95d1d0c2dfc1b018b4a28672e1547f2b55391c9dbdf": { "Name": "test1", "EndpointID": "6b36692de2f89381a53978427ee57066dd35238a326ceccd05bd7ab3fcfb9678", "MacAddress": "", "IPv4Address": "", "IPv6Address": "" } }, "Options": {}, "Labels": {} } ]
可以看到host网络中目前只有一个容器正在使用,就是我们刚刚创建的,但是注意它的MacAddress、IPv4Address、IPv6Address都是空的。此时我们再进入test1容器中查看容器网络情况。
- 容器网络情况
[root@localhost ~]# docker exec -it test1 /bin/sh / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0c:29:f4:16:7e brd ff:ff:ff:ff:ff:ff inet 192.168.0.109/24 brd 192.168.0.255 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::84a4:73f5:46d2:79d3/64 scope link valid_lft forever preferred_lft forever 3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue qlen 1000 link/ether 52:54:00:de:c3:4c brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 qlen 1000 link/ether 52:54:00:de:c3:4c brd ff:ff:ff:ff:ff:ff 5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue link/ether 02:42:87:a3:99:34 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:87ff:fea3:9934/64 scope link valid_lft forever preferred_lft forever 10: br-2d6d1e198a6c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue link/ether 02:42:ae:7a:eb:e7 brd ff:ff:ff:ff:ff:ff inet 172.18.0.1/16 brd 172.18.255.255 scope global br-2d6d1e198a6c valid_lft forever preferred_lft forever inet6 fe80::42:aeff:fe7a:ebe7/64 scope link valid_lft forever preferred_lft forever 16: veth84e019c@if15: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue master br-2d6d1e198a6c link/ether ce:24:54:b5:2e:5c brd ff:ff:ff:ff:ff:ff inet6 fe80::cc24:54ff:feb5:2e5c/64 scope link valid_lft forever preferred_lft forever
容器内的网络很全面,实际上与主机网络做比较后,我们发现它的网络与主机共享同一套网络:
[root@localhost ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:f4:16:7e brd ff:ff:ff:ff:ff:ff inet 192.168.0.109/24 brd 192.168.0.255 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::84a4:73f5:46d2:79d3/64 scope link valid_lft forever preferred_lft forever 3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000 link/ether 52:54:00:de:c3:4c brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000 link/ether 52:54:00:de:c3:4c brd ff:ff:ff:ff:ff:ff 5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:87:a3:99:34 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:87ff:fea3:9934/64 scope link valid_lft forever preferred_lft forever 10: br-2d6d1e198a6c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether 02:42:ae:7a:eb:e7 brd ff:ff:ff:ff:ff:ff inet 172.18.0.1/16 brd 172.18.255.255 scope global br-2d6d1e198a6c valid_lft forever preferred_lft forever inet6 fe80::42:aeff:fe7a:ebe7/64 scope link valid_lft forever preferred_lft forever 16: veth84e019c@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-2d6d1e198a6c state UP link/ether ce:24:54:b5:2e:5c brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::cc24:54ff:feb5:2e5c/64 scope link valid_lft forever preferred_lft forever
这样,host网络与主机共享同一套网络空间命名,这样出现的问题就是端口冲突。
2、none
- 创建none网络容器
[root@localhost ~]# docker run -d --name test1 --network none busybox /bin/sh -c "while true; do sleep 3600; done " a49418c5e24b5019b0931345bf9ade0caaddde2e615f595008f2d91c0eb62958
- 查看none网络
[root@localhost ~]# docker network inspect none [ { "Name": "none", "Id": "d81e2ab77dcf63710f745b2c14d5bee4f46d252abc411bbc42a82d0820fe34d5", "Created": "2019-07-02T22:55:28.453452746+08:00", "Scope": "local", "Driver": "null", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "a49418c5e24b5019b0931345bf9ade0caaddde2e615f595008f2d91c0eb62958": { "Name": "test1", "EndpointID": "881efcbab7ae4f37772a3732183228321d5202d3bd121e71a8bcab02a90f9f72", "MacAddress": "", "IPv4Address": "", "IPv6Address": "" } }, "Options": {}, "Labels": {} } ]
可以看到none网络中刚刚创建的容器也是没有MacAddress、IPv4Address、IPv6Address信息。
- 容器内部网络
[root@localhost ~]# docker exec -it test1 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever failed to resize tty, using default size
可以看到没有多余的网路,这样也就造成了无法通信。
3、总结
目前普遍用bridge网络的情况较多,而host和none网络的情况较少。
来源:https://www.cnblogs.com/shenjianping/p/12243214.html