In the Pyramid web framework, how do I source sensitive settings into development.ini / production.ini from an external file?

Question

I'd like to keep development.ini and production.ini under version control, but for security reason would not want the sqlalchemy.url connection string to be stored, as this would contain the username and password used for the database connection.

What's the canonical way, in Pyramid, of sourcing this setting from an additional external file?

Edit In addition to solution using the environment variable, I came up with this solution after asking around on #pyramid:

def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
# Read db password from config file outside of version control
secret_cfg = ConfigParser()
secret_cfg.read(settings['secrets'])
dbpass = secret_cfg.get("secrets", "dbpass")
settings['sqlalchemy.url'] = settings['connstr'] % (dbpass,)

Answer 1

I looked into this a lot and played with a lot of different approaches. However, Pyramid is so flexible, and the .ini config parser is so minimal in what it does for you, that there doesn't seem to be a de facto answer.

In my scenario, I tried having a production.example.ini in version control at first that got copied on the production server with the details filled in, but this got hairy, as updates to the example didn't get translated to the copy, and so the copy had to be re-created any time a change was made. Also, I started using Heroku, so files not in version control never made it into the deployment.

Then, there's the encrypted config approach. Which, I don't like the paradigm. Imagine a sysadmin being responsible for maintaining the production environment, but he or she is unable to change the location of a database or environment-specific setting without running it back through version control. It's really nice to have the separation between environment and code as much as possible so those changes can be made on the fly without version control revisions.

My ultimate solution was to have some values that looked like this:

[app:main]

sqlalchemy.url = ${SQLALCHEMY_URL}

Then, on the production server, I would set the environment variable SQLALCHEMY_URL to point to the database. This even allowed me to use the same configuration file for staging and production, which is nice.

In my Pyramid init, I just expanded the environment variable value using os.path.expandvars :

sqlalchemy_url = os.path.expandvars(settings.get('sqlalchemy.url'))
engine = create_engine(sqlalchemy_url)

And, if you want to get fancy with it and automatically replace all the environment variables in your settings dictionary, I made this little helper method for my projects:

def expandvars_dict(settings):
    """Expands all environment variables in a settings dictionary."""
    return dict((key, os.path.expandvars(value)) for
                key, value in settings.iteritems())

Use it like this in your main app entry point:

settings = expandvars_dict(settings)

Answer 2

The whole point of the separate ini files in Pyramid is that you do not have to version control all of them and that they can contain different settings for different scenarios (development/production/testing). Your production.ini almost always should not be in the same VCS as your source code.

Answer 3

I found this way for loading secrets from a extra configuration and from the env.

from pyramid.config import Configurator
from paste.deploy import appconfig
from os import path

__all__ = [ "main" ]


def _load_secrets(global_config, settings):
    """ Helper to load secrets from a secrets config and
        from env (in that order).
    """
    if "drawstack.secrets" in settings:
        secrets_config = appconfig('config:' + settings["drawstack.secrets"],
                                    relative_to=path.dirname(global_config['__file__']))

        for k, v in secrets_config.items():
            if k == "here" or k == "__file__":
                continue
            settings[k] = v

    if "ENV_DB_URL" in global_config:
        settings["sqlalchemy.url"] = global_config["ENV_DB_URL"]


def main(global_config, **settings):
    """ This function returns a Pyramid WSGI application.
    """

    _load_secrets(global_config, settings)

    config = Configurator(settings=settings)
    config.include('pyramid_jinja2')
    config.include('.models')
    config.include('.routes')
    config.scan()
    return config.make_wsgi_app()

The code above, will load any variables from the value of the config key drawstack.secrets and after that it tries to load DB_URL from the enviornment.

drawstack.secrets can be relative to the original config file OR absolute.