Mount rexray/ceph volume in multiple containers on Docker swarm

问题

What I have done

I have built a Docker Swarm cluster where I am running containers that have persistent data. To allow the container to move to another host in the event of failure I need resilient shared storage across the swarm. After looking into the various options I have implemented the following:

1. Installed a Ceph Storage Cluster across all nodes of the Swarm and create a RADOS Block Device (RBD). http://docs.ceph.com/docs/master/start/quick-ceph-deploy/

2. Installed Rexray on each node and configure it to use the RBD created above. https://rexray.readthedocs.io/en/latest/user-guide/storage-providers/ceph/

3. Deploy a Docker stack that mounts a volume using the rexray driver e.g.

   version: '3'
   services:
     test-volume:
       image: ubuntu
       volumes:
         - test-volume:/test
   volumes:
     test-volume:
       driver: rexray
   

This solution is working in that I can deploy a stack, simulate a failure on the node that is running then observe the stack restarted on another node with no loss of persistent data.

However, I cannot mount a rexray volume in more than one container. My reason for doing is to use a short lived "backup container" that simply tars the volume to a snapshot backup while the container is still running.

My Question

Can I mount my rexray volumes into a second container?

The second container only needs read access so it can tar the volume to a snapshot backup while keeping the first container running.

回答1:

Unfortunately the answer is no, in this use case rexray volumes cannot be mounted into a second container. Some information below will hopefully assist anyone heading down a similar path:

1. Rexray does not support multiple mounts:

   Today REX-Ray was designed to actually ensure safety among many hosts that could potentially have access to the same host. This means that it forcefully restricts a single volume to only be available to one host at a time. (https://github.com/rexray/rexray/issues/343#issuecomment-198568291)

2. But Rexray does support a feature called pre-emption where:

   ..if a second host does request the volume that he is able to forcefully detach it from the original host first, and then bring it to himself. This would simulate a power-off operation of a host attached to a volume where all bits in memory on original host that have not been flushed down is lost. This would support the Swarm use case with a host that fails, and a container trying to be re-scheduled. (https://github.com/rexray/rexray/issues/343#issuecomment-198568291)

3. However, pre-emption is not supported by the Ceph RBD. (https://rexray.readthedocs.io/en/stable/user-guide/servers/libstorage/#preemption)

来源：https://stackoverflow.com/questions/52731504/mount-rexray-ceph-volume-in-multiple-containers-on-docker-swarm