Heroku: Python dependencies in private repos without storing my password

Question

The Problem

My problem is exactly like How do I install in-house requirements for Python Heroku projects? and How to customize pip's requirements.txt in Heroku on

Answer 1

Create a private PyPI server

If you create your own PyPI server, you can simply list your packages in your requirements.txt file and then store the url for your server (including username and password) in the config variable, PIP_EXTRA_INDEX_URL.

For example:
heroku config:set PIP_EXTRA_INDEX_URL='https://username:password@privateserveraddress.com/simple'

Note that this is the same as using the pip install command line option, --extra-index-url. (See https://pip.pypa.io/en/stable/user_guide/#environment-variables) The primary index url will still be the default (https://pypi.org/simple). This means that pip will first attempt to resolve package names in your requirements file at the default PyPI server, and then try your private server second.

If you need packages in your private server that have the same name as packages in PyPI, then you need the primary index url to be your server and the --extra-index-url option to be the default server's url. You would need to do this if you want to host your own version of an existing package without changing the package name. I haven't tried this, but it currently looks like you would need to to create a fork of heroku's official python buildpack and make a small change to the bin/steps/pip-install file.

The reason pip has access to the PIP_EXTRA_INDEX_URL is because of this block in that file:

    # Set Pip env vars
    # This reads certain environment variables set on the Heroku app config
    # and makes them accessible to the pip install process.
    #
    # PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used.
    if [[ -r "$ENV_DIR/PIP_EXTRA_INDEX_URL" ]]; then
        PIP_EXTRA_INDEX_URL="$(cat "$ENV_DIR/PIP_EXTRA_INDEX_URL")"
        export PIP_EXTRA_INDEX_URL
        mcount "buildvar.PIP_EXTRA_INDEX_URL"
    fi

Code like this is necessary to read config variables in buildpacks (see https://devcenter.heroku.com/articles/buildpack-api#buildpack-api), but you should be able to simply duplicate this codeblock, replacing PIP_EXTRA_INDEX_URL with PIP_INDEX_URL. Then set PIP_INDEX_URL to your private server's url and PIP_EXTRA_INDEX_URL to the default PyPI url.

If you are using another source instead of a private PyPI server, such as github, and simply need a way to avoid hardcoding a username and password in your requirements.txt file, then also note that you can use environment variables in requirements.txt (see https://pip.pypa.io/en/stable/reference/pip_install/#using-environment-variables). You would just have to export them in bin/steps/pip-install as you would for PIP_INDEX_URL.