Evaluate statements from within Python logging YAML config file

Question

Consider the following snippet of a Python logging YAML config file:

version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  logfile:
    class: logging.handlers.TimedRotatingFileHandler
    level: DEBUG
    filename: some_fancy_import_name.generate_filename_called_error
    backupCount: 5
    formatter: simple

I would like to load this YAML config file this way:

with open('logging.yaml', 'r') as fd:
    config = yaml.safe_load(fd.read())
logging.config.dictConfig(config)

Take special notice of the filename to which the handler should write logs. In normal Python code, I would expect some_fancy_import_name.generate_filename_called_errorlog to generate the string 'error.log' . All in all, I would like to say that this logging handler should write to the file 'error.log' in the current directory.

However, as it turns out, this is not the case. When I look at the current directory, I see a file named 'some_fancy_import_name.generate_filename_called_errorlog' .

Why go through all this trouble?

I would like filename to be programmatically determined. I have successfully tried configuring logging using normal Python scripting this way:

# fancy import name
from os import environ as env

# Programmatically determine filename path
log_location = env.get('OPENSHIFT_LOG_DIR', '.')
log_filename = os.path.join(log_location, 'error')
handler = logging.handlers.TimedRotatingFileHandler(log_filename)

See how the log_filename path was inferred from environment variables.

I would like to translate this to a YAML config file. Is it possible?

Perhaps I might need to dig through the dict produced by yaml.safe_load(fd.read()) and do some eval() stuff?

Answer 1

You can add a custom constructor and mark the value with a special tag, so your constructor gets executed when loading it:

import yaml

def eval_constructor(loader, node):
  return eval(loader.construct_scalar(node))

yaml.add_constructor(u'!eval', eval_constructor)

some_value = '123'

config = yaml.load("""
version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  logfile:
    class: logging.handlers.TimedRotatingFileHandler
    level: DEBUG
    filename: !eval some_value
    backupCount: 5
    formatter: simple
""")

print config['handlers']['logfile']['filename']

This prints 123 , since the value some_value has the tag !eval , and therefore is loaded with eval_constructor .

Be aware of the security implications of eval ing configuration data. Arbitrary Python code can be executed by writing it into the YAML file!

Answer 2

Solution

Thanks to flyx's answer , this is how I did it:

import logging
import yaml
from os import environ as env

def constructor_logfilename(loader, node):
    value = loader.construct_scalar(node)
    return os.path.join(env.get('OPENSHIFT_LOG_DIR', '.'), value)

yaml.add_constructor(u'!logfilename', constructor_logfilename)
with open('logging.yaml', 'r') as fd:
    config = yaml.load(fd.read())
logging.config.dictConfig(config)

In the logging.yaml file, here's the important snippet:

...
filename: !logfilename error.log
...

Answer 3

I just now had this issue come up myself and I approached it in a much more boring way. I left out the filename attribute of the logfile handler from my yaml file and added it to the config dictionary created by yaml.load like this:

with open('logging.yaml', 'r') as fd:
    config = yaml.load(fd.read())
config['handlers']['logfile']['filename'] = generate_logfile_name(...)
logging.config.dictConfig(config)