How to auto deploying git repositories with submodules on AWS?

Question

I have a submodule in my git repository and my directory structure is like,

app
  -- folder1
  -- folder2
  -- submodule @5855

I have deployed

Answer 1

While @MattBucci answer works, it has the caveat that you can only pull a specific branch, and not the specific commit that the submodule is using.

In order to handle that case, which is likely when using submodules, there are multiple things that needs to be done:

1) Create a git pre-commit hook with the following content:

#!/bin/bash

#   This file is used in post-commit hook
#   if .commit exists you know a commit has just taken place but a post-commit hasn't run yet
#
touch .commit

If you already have one, you can add that line at the beginning.

2) Create a git post-commit hook with the following contnet:

#!/bin/bash


DIR=$(git rev-parse --show-toplevel);

if [[ -e $DIR/.commit ]]; then
    echo "Generating submodule integrity file"
    rm .commit

    SUBMODULE_TRACKING_FILE=$DIR/.submodule-hash
    MODULE_DIR=module
    #   Get submodule hash, this will be used by AWS Code Build to pull the correct version.
    #   AWS Code Build does not support git submodules at the moment
    #   https://forums.aws.amazon.com/thread.jspa?messageID=764680#764680
    git ls-tree $(git symbolic-ref --short HEAD) $MODULE_DIR/ | awk '{ print $3 }' > $SUBMODULE_TRACKING_FILE

    git add $SUBMODULE_TRACKING_FILE
    git commit --amend -C HEAD --no-verify
fi

exit 0

This hook will put the current commit hash into .submodule-hash file, this file needs to be committed to version control.

3) Go to your AWS Code build project

Developer Tools > CodeBuild > Build projects > YOUR_PROJECT > Edit Environment

Add an environment variable called: GIT_KEY, and the value will be the ssh key base 64 encoded. (Without line breaks, otherwise it won't work).

You can convert it online, or use any tool or programming language.

4) On your buildspec.yml add a pre_build script.

version: 0.2

phases:
  pre_build:
    commands:
      - bash build/aws-pre-build.sh
...

5) Create build/aws-pre-build.sh with the following content:

#!/bin/bash

set -e

#   Get root path
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd .. && pwd )"

MODULE_HASH=$(cat $DIR/.submodule-hash);
GIT_HOST=bitbucket.org
MODULE_DIR=module
REPO=user/repo.git


if [[ ! -d ~/.ssh ]]; then
    mkdir ~/.ssh
fi

if [[ ! -f ~/.ssh/known_hosts ]]; then
    touch ~/.ssh/known_hosts
fi

#   Base64 decode private key, and save it to ~/.ssh/git
echo "- Adding git private key"

echo $GIT_KEY | base64 -d > ~/.ssh/git

#   Add correct permissions to key
chmod 600 ~/.ssh/git

#   Add $GIT_HOST to ssh config
echo "- Adding ssh config file"

cat > ~/.ssh/config <<_EOF_
Host $GIT_HOST
    User git
    IdentityFile ~/.ssh/git
    IdentitiesOnly yes
_EOF_

#   Check if host is present in known_hosts
echo "- Checking $GIT_HOST in known_hosts"

if ! ssh-keygen -F $GIT_HOST > /dev/null; then
    echo "- Adding $GIT_HOST to known hosts"
    ssh-keyscan -t rsa $GIT_HOST >> ~/.ssh/known_hosts
fi

#   AWS Code build does not send submodules, remove the empty folder
rm -rf $MODULE_DIR

# Clone submodule in the right folder
git clone git@$GIT_HOST:$REPO $MODULE_DIR

# cd to submodule
cd $DIR/$MODULE_DIR

# Checkout the right commit
echo "- Checking out $MODULE_HASH"

git checkout $MODULE_HASH

---

Extras

If you have an extra step before going to AWS Code Build, something like bitbucket pipelines or similar, you can check that the actual git submodule hash, matches the hash from the generated file: .submodule-hash.

If it does not match, it means who ever pushed, didn't have the git hook.

#!/bin/bash

$MODULE_DIR=module

echo "- Checking submodules integrity"

SUBMODULE_TRACKING_FILE=.submodule-hash


#   Check submodule hash, this will be used by AWS Code Build to pull the correct version.
#   AWS Code Build does not support git submodules at the moment
#   https://forums.aws.amazon.com/thread.jspa?messageID=764680#764680

#   Git submodule actual hash
SUBMODULE_HASH=$(git ls-tree $(git symbolic-ref --short HEAD) $MODULE_DIR/ | awk '{ print $3 }')

if [[ ! -e $SUBMODULE_TRACKING_FILE ]]; then

    echo "ERROR: $SUBMODULE_TRACKING_FILE file not found."
    submoduleError

    exit 1
fi

#   Custom submodule hash - The is used by AWS Code Build
SUBMODULE_TRACKING_FILE_HASH=$(cat $SUBMODULE_TRACKING_FILE)

if [[ "$SUBMODULE_TRACKING_FILE_HASH" != "$SUBMODULE_HASH"  ]]; then

    echo "ERROR: $SUBMODULE_TRACKING_FILE file content does not match submodule hash: $SUBMODULE_HASH"

    echo -e "\tYou should have pre-commit && post-commit hook enabled or update $SUBMODULE_TRACKING_FILE manually:"
    echo -e "\tcmd: git ls-tree $(git symbolic-ref --short HEAD) $MODULE_DIR/ | awk '{ print \$3 }' > $SUBMODULE_TRACKING_FILE"

    exit 1
fi

NOTE: You can also create that file on the pipeline before AWS Code Build, create a commit, tag it, and push it so the AWS Code Build pipeline begins.

git ls-tree $(git symbolic-ref --short HEAD) module/ | awk '{ print \$3 }' > .submodule-hash

Answer 2

Edit: Codebuild now has a "submodules" flag https://docs.aws.amazon.com/codebuild/latest/APIReference/API_GitSubmodulesConfig.html

Here's what worked for me

We're going to reinitialize the git repository and then trigger a submodule clone during the build phase of our deploy, essentially patching in support for submodules in codepipeline / codebuild

• Generate a new SSH key for your github account, if using an organization you may want to create a deploy user
• Store this ssh key in your aws parameter store using aws ssm put-parameter --name build_ssh_key --type String --value "$(cat id_rsa)" ideally use SecureString instead of String but the guide I was following simply used string so I'm not sure if the commandline will require any extra params
• Go into IAM and grant your CodePipeline user read access to your paramstore, I just granted read access to SSM

Then make your buildspec.yml look like the following:

version: 0.2

env:
  parameter-store:
    build_ssh_key: "build_ssh_key"

phases:
  install:
    commands:
      - mkdir -p ~/.ssh
      - echo "$build_ssh_key" > ~/.ssh/id_rsa
      - chmod 600 ~/.ssh/id_rsa
      - ssh-keygen -F github.com || ssh-keyscan github.com >>~/.ssh/known_hosts
      - git config --global url."git@github.com:".insteadOf "https://github.com/"
      - git init
      - git remote add origin <Your Repo url here using the git protocol>
      - git fetch
      - git checkout -t origin/master
      - git submodule init
      - git submodule update --recursive
  build:
    commands:
      - echo '...replace with real build commands...'

artifacts:
  files:
    - '**/*'