Is it possible to store the alembic connect string outside of alembic.ini?

Question

I\'m using Alembic with SQL Alchemy. With SQL Alchemy, I tend to follow a pattern where I don\'t store the connect string with the versioned code. Instead I have file secr

Answer 1

I had the very same problem yesterday and found a following solution to work. I do the following in alembic/env.py:

# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config

# this will overwrite the ini-file sqlalchemy.url path
# with the path given in the config of the main code
import config as ems_config
config.set_main_option('sqlalchemy.url', ems_config.config.get('sql', 'database'))

ems_config is an external module that holds my configuration data.

config.set_main_option(...) essentially overwrites the sqlalchemy.url key in the [alembic] section of the alembic.ini file. In my configuration I simply leave it black.

Answer 2

Another solution is to create a template alembic.ini.dist file and to track it with your versionned code, while ignoring alembic.ini in your VCS.

Do not add any confidential information in alembic.ini.dist:

sqlalchemy.url = ...

When deploying your code to a platform, copy alembic.ini.dist to alembic.ini (this one won't be tracked by your VCS) and modify alembic.ini with the platform's credentials.

Answer 3

So what appears to work is reimplementing engine creation in env.py, which is apparently a place for doing this kind of customizing Instead of using the sqlalchemy connect string in the ini:

engine = engine_from_config(
            config.get_section(config.config_ini_section),
            prefix='sqlalchemy.',
           poolclass=pool.NullPool)

You can replace and specify your own engine configuration:

import store
engine = store.engine

Indeed the docs seems to imply this is ok:

sqlalchemy.url - A URL to connect to the database via SQLAlchemy. This key is in fact only referenced within the env.py file that is specific to the “generic” configuration; a file that can be customized by the developer. A multiple database configuration may respond to multiple keys here, or may reference other sections of the file.

Answer 4

I was looking for a while how to manage this for mutli-databases

Here is what I did. I have two databases : logs and ohlc

According to the doc, I have setup the alembic like that

alembic init --template multidb

alembic.ini

databases = logs, ohlc
[logs]
sqlalchemy.url = postgresql://botcrypto:botcrypto@localhost/logs
[ohlc]
sqlalchemy.url = postgresql://botcrypto:botcrypto@localhost/ohlc

env.py

[...]
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config

# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)
logger = logging.getLogger('alembic.env')

# overwrite alembic.ini db urls from the config file
settings_path = os.environ.get('SETTINGS')
if settings_path:
    with open(settings_path) as fd:
        settings = conf.load(fd, context=os.environ) # loads the config.yml
    config.set_section_option("ohlc", "sqlalchemy.url", settings["databases"]["ohlc"])
    config.set_section_option("logs", "sqlalchemy.url", settings["databases"]["logs"])
else:
    logger.warning('Environment variable SETTINGS missing - use default alembic.ini configuration')
[...]

config.yml

databases:
    logs: postgresql://botcrypto:botcrypto@127.0.0.1:5432/logs
    ohlc: postgresql://botcrypto:botcrypto@127.0.0.1:5432/ohlc

usage

SETTINGS=config.yml alembic upgrade head

Hope it can helps !

Answer 5

I was bumping into this problem as well since we're running our migrations from our local machines. My solution is to put environment sections in the alembic.ini which stores the database config (minus the credentials):

[local]
host = localhost
db = dbname

[test]
host = x.x.x.x
db = dbname

[prod]
host = x.x.x.x
db = dbname

Then I put the following in the env.py so the user can pick their environment and be prompted for the credentials:

from alembic import context
from getpass import getpass

...

envs = ['local', 'test', 'prod']

print('Warning: Do not commit your database credentials to source control!')
print(f'Available migration environments: {", ".join(envs)}')

env = input('Environment: ')

if env not in envs:
    print(f'{env} is not a valid environment')
    exit(0)

env_config = context.config.get_section(env)
host = env_config['host']
db = env_config['db']

username = input('Username: ')
password = getpass()
connection_string = f'postgresql://{username}:{password}@{host}/{db}'

context.config.set_main_option('sqlalchemy.url', connection_string)

You should store your credentials in a password manager that the whole team has access to, or whatever config/secret store you have available. Though, with this approach the password is exposed to your local clip board - an even better approach would be to have env.py directly connect to your config/secret store API and pull out the username/password directly but this adds a third party dependency.

Answer 6

As Doug T. said you can edit env.py to provide URL from somewhere else than ini file. Instead of creating new engine you can pass an additional url argument to the engine_from_config function (kwargs are later merged to options taken from ini file). In that case you could e.g. store encrypted password in ini file and decrypt it in runtime by passphrase stored in ENV variable.

connectable = engine_from_config(                 
    config.get_section(config.config_ini_section),
    prefix='sqlalchemy.',                         
    poolclass=pool.NullPool,                      
    url=some_decrypted_endpoint)