Airflow inside docker running a docker container

Question

I have airflow running on an EC2 instance, and I am scheduling some tasks that spin up a docker container. How do I do that? Do I need to install docker on my airflow container?

Answer 1

You can spin up docker containers from your airflow docker container by attaching volumes to your container.

Example:

docker run -v /var/run/docker.sock:/var/run/docker.sock:ro -v /path/to/bin/docker:/bin/docker:ro your_airflow_image

You may also need to attach some libraries required by docker. This depends on the system you are running Docker on. Just read the error messages you get when running a docker command inside the container, it will indicate you what you need to attach.

Your airflow container will then have full access to Docker running on the host. So if you launch docker containers, they will run on the host running the airflow container.