Docker Swarm container with MACVLAN network gets wrong gateway - no internet access

问题

I need a service within a Docker Swarm stack which has one additional interface based on a macvlan network. This is cause the JBoss Cluster in this service needs to communicate via IP multicast, which is currently not supported in overlay networks.

I have created the macvlan network like that:

# Worker 1:
docker network create --config-only --subnet 10.140.0.0/16 -o parent=ens224.800 --ip-range 10.140.1.0/24 swarm-multicast-config-only

# Worker 2:
docker network create --config-only --subnet 10.140.0.0/16 -o parent=ens224.800 --ip-range 10.140.2.0/24 swarm-multicast-config-only

# Worker 3:
docker network create --config-only --subnet 10.140.0.0/16 -o parent=ens224.800 --ip-range 10.140.3.0/24 swarm-multicast-config-only

# Master:
docker network create -d macvlan --scope swarm --internal --config-from swarm-multicast-config-only swarm-multicast

Multicast works perfectly fine like that, the cluster forms.

But: As soon as I assign this macvlan network to one of my containers, this container can no longer access the internet. All containers without the macvlan network work perfectly fine.

Here's my stack file:

version: '3.3'
services:
  ### Backend ###
  petshop-backend:
    image: com-registry.xxx.local/petshop-backend:100
    extra_hosts:
      - "petshop-db:10.164.210.214"
    networks:
      - backend
      - external_access
    deploy:
      mode: replicated
      replicas: 3

  ### USER INTERFACE ###
  petshop-ui:
    image: com-registry.xxx.local/petshop-ui:107
    networks:
      external_access:
      backend:
      swarm-multicast:
        aliases:
          - ui-multicast
    ports:
      - "1002:8080"
    deploy:
      mode: replicated
      replicas: 3


networks:
  external_access:
    driver: overlay
    internal: false
  backend:
    driver: overlay
    internal: true
  swarm-multicast:
    external: true

How can I enable the containers of petshop-ui to access the internet? They get a default gateway of 10.140.1.0, wich is from the range of the macvlan network, but does not exist. Here's the routing table of one of the petshop-ui containers:

[root@f477c7cb8048 /]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use     Iface
0.0.0.0         10.140.1.0      0.0.0.0         UG    0      0        0 eth2
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth4
10.0.1.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.140.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth2
10.255.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth3

Containers with working internet access, e.g. petshop-backend have 172.18.0.1 as default gateway. Here's such a routing table:

[root@ddb42ef836f3 /]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use     Iface
0.0.0.0         172.18.0.1      0.0.0.0         UG    0      0        0 eth2
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.0.1.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth2

回答1:

You need to change

networks:
  external_access:
    driver: overlay
    internal: false
  backend:
    driver: overlay
    internal: true
  swarm-multicast:
    external: true

to

networks:
  backend:
    driver: overlay
    internal: true
  swarm-multicast:
    external: true
  external_access:
    driver: overlay
    internal: false

Currently it seems the last network attached takes over the gateway route. There is a open issue for the same

https://github.com/moby/moby/issues/20179

来源：https://stackoverflow.com/questions/46511409/docker-swarm-container-with-macvlan-network-gets-wrong-gateway-no-internet-acc