howto: elastic beanstalk + deploy docker + graceful shutdown

Question

Hi great people of stackoverflow,

Were hosting a docker container on EB with an nodejs based code running on it. When redeploying our docker container we\'d like the old

Answer 1

Self answering as I've found a solution that works for us:

tl;dr: use .ebextensions scripts to run your script before 01flip, your script will make sure a graceful shutdown of whatevers inside the docker takes place

first, your app (or whatever your'e running in docker) has to be able to catch a signal, SIGINT for example, and shutdown gracefully upon it.
this is totally unrelated to Docker, you can test it running wherever (locally for example) There is a lot of info about getting this kind of behaviour done for different kind of apps on the net (be it ruby, node.js etc...)

Second, your EB/Docker based project can have a .ebextensions folder that holds all kinda of scripts to execute while deploying. we put 2 custom scripts into it, gracefulshutdown_01.config and gracefulshutdown_02.config file that looks something like this:

# gracefulshutdown_01.config
commands:
  backup-original-flip-hook:
    command: cp -f /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh /opt/elasticbeanstalk/hooks/appdeploy/01flip.sh.bak
    test: '[ ! -f /opt/elasticbeanstalk/hooks/appdeploy/01flip.sh.bak ]'
  cleanup-custom-hooks:
    command: rm -f 05gracefulshutdown.sh
    cwd: /opt/elasticbeanstalk/hooks/appdeploy/enact
    ignoreErrors: true

and:

# gracefulshutdown_02.config
commands:
  reorder-original-flip-hook:
    command: mv /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh /opt/elasticbeanstalk/hooks/appdeploy/enact/10flip.sh
    test: '[ -f /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh ]'

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/enact/05gracefulshutdown.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/bin/sh

      # find currently running docker
          EB_CONFIG_DOCKER_CURRENT_APP_FILE=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_file)
      EB_CONFIG_DOCKER_CURRENT_APP=""

      if [ -f $EB_CONFIG_DOCKER_CURRENT_APP_FILE ]; then
        EB_CONFIG_DOCKER_CURRENT_APP=`cat $EB_CONFIG_DOCKER_CURRENT_APP_FILE | cut -c 1-12`
        echo "Graceful shutdown on app container: $EB_CONFIG_DOCKER_CURRENT_APP"
      else
        echo "NO CURRENT APP TO GRACEFUL SHUTDOWN FOUND"
        exit 0
      fi

      # give graceful kill command to all running .js files (not stats!!)
      docker exec $EB_CONFIG_DOCKER_CURRENT_APP sh -c "ps x -o pid,command | grep -E 'workers' | grep -v -E 'forever|grep' " |  awk '{print $1}' | xargs docker exec $EB_CONFIG_DOCKER_CURRENT_APP kill -s SIGINT
      echo "sent kill signals"

      # wait (max 5 mins) until processes are done and terminate themselves
      TRIES=100
      until [ $TRIES -eq 0 ]; do
        PIDS=`docker exec $EB_CONFIG_DOCKER_CURRENT_APP sh -c "ps x -o pid,command | grep -E 'workers' | grep -v -E 'forever|grep' " | awk '{print $1}' | cat`
        echo TRIES $TRIES PIDS $PIDS
        if [ -z "$PIDS" ]; then
          echo "finished graceful shutdown of docker $EB_CONFIG_DOCKER_CURRENT_APP"
          exit 0
        else
          let TRIES-=1
          sleep 3
        fi
      done

      echo "failed to graceful shutdown, please investigate manually"
      exit 1

gracefulshutdown_01.config is a small util that backups the original flip01 and deletes (if exists) our custom script.

gracefulshutdown_02.config is where the magic happens. it creates a 05gracefulshutdown enact script and makes sure flip will happen afterwards by renaming it to 10flip.

05gracefulshutdown, the custom script, does this basically:

• find current running docker
• find all processes that need to be sent a SIGINT (for us its processes with 'workers' in its name
• send a sigint to the above processes
• loop:
• check if processes from before were killed
• continue looping for an amount of tries
• if tries are over, exit with status "1" and dont continue to 10flip, manual interference is needed.

this assumes you only have 1 docker running on the machine, and that you are able to manually hop on to check whats wrong in the case it fails (for us never happened yet).
I imagine it can also be improved in many ways, so have fun.