Docker container keeps growing

Question

I have a pyhton script that on a loop

1. Downloads video chunks from AWS S3 to /filename
2. Sorts files in order and concats them.
3. Uploads entire

Answer 1

I would suggest you to use volumes, and mount these volumes in your containers. Changes on volumes are instantaneous, as opposed to changes made to the containers filesystem (which is not really removed until you delete the container).

Have a look at the docs here

Answer 2

If you're downloading the image to a directory NOT volumed mapped from the host, the docker container will not release the used disk space until the container is removed--anything done in the container is ephemeral, but the HOST doesn't know the state of what's going on inside the container.

In this sense it's a lot like a virtual machine image, backed by a file that just grows as needed, but never shrinks. Docker has a directory for a running container tracking changes. On the host you can find the files backing the running container in /var/lib/docker/containers/<id>

If you need your containers to share disk space, I'd recommend you map a shared volume from the host into each docker container images to share.

Try the following

 docker run -ti -v /host/dir:/container/dir ubuntu bash

The above would run the ubuntu image in terminal interactive mode and mounting the host's directory /host/dir inside the running container. Anything the container writes to /container/dir will appear in the hosts /host/dir and any other containers mounting it will see the changes as well.

Just remember anything done in the shared volume is seen by all containers that mount it, so be careful when adding and deleting files/directories from it!