Heroku: Python dependencies in private repos without storing my password

Question

The Problem

My problem is exactly like How do I install in-house requirements for Python Heroku projects? and How to customize pip's requirements.txt in Heroku on

Answer 1

I faced the same problem. Like you, I am amazed how difficult it is to find good documentation on how to install private dependency (whatever the language and the service used).

Because this is not a main concern of service providers, I now try a systematic approach relying as few as possible on idiosyncratic features. I try to find the easier solution for each of these steps:

• Pass the credentials to the build environment using a secure channel. For python, use an environment variable containing a SSH key as a base64 string. For js, same with the npm token.
• Configure the build process to use these credentials. In the best case it involves configuring ssh to use a deploy key. Otherwise it can as basic as cloning the dependency for later use. For your specific case with python and heroku, you can use the hook 'pre_compile'.

I detailed the process for my future self here: https://gist.github.com/michelbl/a6163522d95540cf0c8b6667bd35d5f5

---

I need to give access to a private dependency. It can happen for continuous integration or deployment.

Here we use python and github, using the services CircleCI and Heroku. However, the principles applies everywhere.

What is a deploy key?

See https://developer.github.com/v3/guides/managing-deploy-keys/

There are 4 ways of granting access to a private dependency, but deploy keys are a good compromise in term of security and ease of use for projects that do not require too many dependencies (in that case, prefer a machine user). In any case, do not use username/password of a developer account or oauth token as they do not provide privilege limitation.

Create a deploy key:

ssh-keygen -t rsa -b 4096 -C "myself@my_company.com"

Give the public part to gihub.

Give the private part to the service needing access. See below.

General strategy

Whatever the service or the technology that I use, the goal is to access the git repo using ssh, using the deploy key.

Obviously, I do not want to put the deploy key in the repo. But most services (CI, deployment) provide a way to set protected environment variables that can be used at build time. The key can be encoded using base64:

cat deploy-key | base64
cat deploy-key.pub | base64

Most services also provide a way to tailor the build procedure. This is needed to configure ssh to use the deploy key.

CircleCI

Set the deploy key using env variables, encode with base64.

In config.yml, add a step:

echo $DEPLOY_KEY_PRIVATE | base64 --decode > ~/.ssh/deploy-key
chmod 400 ~/.ssh/deploy-key
echo $DEPLOY_KEY_PUBLIC | base64 --decode > ~/.ssh/deploy-key.pub
ssh-add ~/.ssh/deploy-key

# Run this to check which private key is used. If the checkout key is used,
# github replies "Hi my_org/my_package". If the deploy key is used as wished,
# github replies "Hi my_org/my_dependency".
#ssh -i ~/.ssh/deploy-key -T git@github.com || true

# Now pip connects to git+ssh using the deploy key
export GIT_SSH_COMMAND="ssh -i ~/.ssh/deploy-key"

pip install -r requirements.txt

requirements.txt can be something like:

# The purpose of this file is to install the private dependency *before*
# setup.py is run.

# Be sure ssh is configured to use a ssh key with read permission to the repo.
git+ssh://git@github.com/my_org/my_dependency@1.0.10

# Run setup.py. The private dependency is already installed with the good
# version so pip doesn't try to fetch it from PyPI.
--editable .

and setup.py does not care about the dependency beeing private:

from distutils.core import setup

setup(
    name='my_package',
    version='1.0',
    packages=[
        'my_package',
    ],
    install_requires=[
        # Beware, the following package is a private dependency.
        # Python provides several way to install private dependencies, none
        # are really satisfactory.
        # 1. Use dependency_links / --process-dependency-links. Good luck with
        #    that!
        # 2. Maintain a private package repository. Good luck with that!
        # 3. Install the private dependency separately before setup.py is run.
        #    This is now the prefered way. Be sure that ssh is properly
        #    configured to use a ssh key with read permission to the github repo
        #    of the private dependency, then run:
        #    `pip install -r requirements.txt`
        'my_dependency==1.0.10',
        ... # my normal dependencies
        'unidecode==1.0.22',
        'uwsgi==2.0.15',
        'nose==1.3.7', # tests
        'flake8==3.5.0', # style
    ],
)

Heroku

For python, there is no need to write a custom buildpack. First, set the deploy key using env variables, encode with base64.

Then add the hook bin/pre_compile:

# This script configures ssh on Heroku to use the deploy key.
# This is needed to install private dependencies.
#
# Note that this does not work with Heroku review apps. Indeed review apps can
# inherits env variables from their parents, but they access their values after
# the build. You would need a way to pass the ssh key to this script another
# way.
#
# See also
# * https://stackoverflow.com/questions/21297755/heroku-python-dependencies-in-private-repos-without-storing-my-password#
# * https://github.com/bjeanes/ssh-private-key-buildpack

# Ensure we have an ssh folder
if [ ! -d ~/.ssh ]; then
  mkdir -p ~/.ssh
  chmod 700 ~/.ssh
fi

# Create the key files
cat $ENV_DIR/DEPLOY_KEY | base64 --decode > ~/.ssh/deploy-key
chmod 400 ~/.ssh/deploy-key
cat $ENV_DIR/DEPLOY_KEY | base64 --decode > ~/.ssh/deploy-key.pub
#ssh-add ~/.ssh/deploy-key

# If you want to disable host verification, you could use that.
#ssh -oStrictHostKeyChecking=no -T git@github.com 2>&1

# Run that if you want to check that ssh uses the correct key.
#ssh -i ~/.ssh/deploy-key -T git@github.com || true

# Configure ssh to use the correct deploy key when connecting to github.
# Disables host verification.
echo -e "Host github.com\n"\
        "  IdentityFile ~/.ssh/deploy-key\n"\
        "  IdentitiesOnly yes\n"\
        "  UserKnownHostsFile=/dev/null\n"\
        "  StrictHostKeyChecking no"\
        >> ~/.ssh/config

# Unfortunately this does not seem to work.
#export GIT_SSH_COMMAND="ssh -i ~/.ssh/deploy-key"

# The vanilla python buildpack can now install all the dependencies in
# requirement.txt