Repetitive wrapper functions for logging

Question

I am writing a script that needs to use a class from an external library, do some operations on instances of that class, and then repeat with some more instances.

Something like this:

import some_library

work_queue = get_items()

for item in work_queue:
  some_object = some_library.SomeClass(item)
  operation_1(some_object)
  # ...
  operation_N(some_object)

However, each of the operations in the loop body can raise some different exceptions. When these happen I need to log them and skip to the next item. If they raise some unexpected exception I need to log that before crashing.

I could catch all the exceptions in the main loop, but that would obscure what it does. So I find myself writing a bunch of wrapper functions that all look kind of similar:

def wrapper_op1(some_object):
  try:
    some_object.method_1()
  except (some_library.SomeOtherError, ValueError) as error_message:
    logger.error("Op1 error on {}".format(some_object.friendly_name))
    return False
  except Exception as error_message:
    logger.error("Unknown error during op1 on {} - crashing: {}".format(some_object.friendly_name, error_message))
    raise
  else:
    return True

# Notice there is a different tuple of anticipated exceptions
# and the message formatting is different
def wrapper_opN(some_object):
  try:
    some_function(some_object.some_attr)
  except (RuntimeError, AttributeError) as error_message:
    logger.error("OpN error on {} with {}".format(some_object.friendly_name, some_object.some_attr, error_message))
    return False
  except Exception as error_message:
    logger.error("Unknown error during opN on {} with {} - crashing: {}".(some_object.friendly_name, some_object.some_attr, error_message))
    raise
  else:
    return True

And modifying my main loop to be:

for item in work_queue:
  some_object = some_library.SomeClass(item)
  if not wrapper_op1(some_object):
    continue
  # ...
  if not wrapper_opN(some_object):
    continue

This does the job, but it feels like a lot of copy and paste programming with the wrappers. What would be great is to write a decorator function that could do all that try...except...else stuff so I could do:

@ logged_call(known_exception, known_error_message, unknown_error_message)
def wrapper_op1(some_object):
  some_object.method_1()

The wrapper would return True if the operation succeeds, catch the known exceptions and log with a specified format, and catch any unknown exceptions for logging before re-raising.

However, I can't seem to fathom how to make the error messages work - I can do it with fixed strings:

def logged_call(known_exceptions, s_err, s_fatal):
  def decorate(f):
    @wraps(f)
    def wrapper(*args, **kwargs):
      try:
        f(*args, **kwargs)
      # How to get parameters from args in the message?
      except known_exceptions as error:
        print(s_err.format(error))
        return False
      except Exception as error:
        print(s_fatal.format(error))
        raise
      else:
        return True
    return wrapper
  return decorate

However, my error messages need to get attributes that belong to the decorated function.

Is there some Pythonic way to make this work? Or a different pattern to be using when dealing with might-fail-in-known-ways functions?

Answer 1

The below tryblock function can be used for this purpose, to implement your logged_call concept and reduce the total amount of code, assuming you have enough checks to overcome the decorator implementation. Folks not used to functional programming in Python may actually find this more difficult to understand than simply writing out the try blocks as you have done. Simplicity, like so many things, is in the eye of the beholder.

Python 2.7 using no imports. This uses the exec statement to create a customized try block that works in a functional pattern.

def tryblock(tryf, *catchclauses, **otherclauses):
  u'return a general try-catch-else-finally block as a function'
  elsef = otherclauses.get('elsef', None)
  finallyf = otherclauses.get('finallyf', None)
  namespace = {'tryf': tryf, 'elsef': elsef, 'finallyf': finallyf, 'func': []}
  for pair in enumerate(catchclauses):
    namespace['e%s' % (pair[0],)] = pair[1][0]
    namespace['f%s' % (pair[0],)] = pair[1][1]
  source = []
  add = lambda indent, line: source.append(' ' * indent + line)
  add(0, 'def block(*args, **kwargs):')
  add(1,   "u'generated function executing a try block'")
  add(1,   'try:')
  add(2,     '%stryf(*args, **kwargs)' % ('return ' if otherclauses.get('returnbody', elsef is None) else '',))
  for index in xrange(len(catchclauses)):
    add(1, 'except e%s as ex:' % (index,))
    add(2,   'return f%s(ex, *args, **kwargs)' % (index,))
  if elsef is not None:
    add(1,   'else:')
    add(2,     'return elsef(*args, **kwargs)')
  if finallyf is not None:
    add(1,   'finally:')
    add(2,     '%sfinallyf(*args, **kwargs)' % ('return ' if otherclauses.get('returnfinally', False) else '',))
  add(0, 'func.append(block)')
  exec '\n'.join(source) in namespace
  return namespace['func'][0]

This tryblock function is general enough to go into a common library, as it is not specific to the logic of your checks. Add to it your logged_call decorator, implemented as (one import here):

import functools
resultof = lambda func: func()  # @ token must be followed by an identifier
@resultof
def logged_call():
  truism = lambda *args, **kwargs: True
  def raisef(ex, *args, **kwargs):
    raise ex
  def impl(exlist, err, fatal):
    return lambda func: \
      functools.wraps(func)(tryblock(func,
          (exlist, lambda ex, *args, **kwargs: err(ex, *args, **kwargs) and False),
          (Exception, lambda ex, *args, **kwargs: fatal(ex, *args, **kwargs) and raisef(ex))),
          elsef=truism)
  return impl  # impl therefore becomes logged_call

Using logged_call as implemented, your two sanity checks look like:

op1check = logged_call((some_library.SomeOtherError, ValueError),
    lambda _, obj: logger.error("Op1 error on {}".format(obj.friendly_name)),
    lambda ex, obj: logger.error("Unknown error during op1 on {} - crashing: {}".format(obj.friendly_name, ex.message)))

opNcheck = logged_call((RuntimeError, AttributeError),
    lambda ex, obj: logger.error("OpN error on {} with {}".format(obj.friendly_name, obj.some_attr, ex.message)),
    lambda ex, obj: logger.error("Unknown error during opN on {} with {} - crashing: {}".format(obj.friendly_name, obj.some_attr, ex.message)))

@op1check
def wrapper_op1(obj):
  return obj.method_1()

@opNcheck
def wrapper_opN(obj):
  return some_function(obj.some_attr)

Neglecting blank lines, this is more compact than your original code by 10 lines, though at the sunk cost of the tryblock and logged_call implementations; whether it is now more readable is a matter of opinion.

You also have the option to define logged_call itself and all distinct decorators derived from it in a separate module, if that's sensible for your code; and therefore to use each derived decorator multiple times.

You may also find more of the logic structure that you can factor into logged_call by tweaking the actual checks.

But in the worst case, where each check has logic that no other does, you may find that it's more readable to just write out each one like you have already. It really depends.

For completeness, here's a unit test for the tryblock function:

import examplemodule as ex
from unittest import TestCase
class TestTryblock(TestCase):
  def test_tryblock(self):
    def tryf(a, b):
      if a % 2 == 0:
        raise ValueError
      return a + b
    def proc_ve(ex, a, b):
      self.assertIsInstance(ex, ValueError)
      if a % 3 == 0:
        raise ValueError
      return a + b + 10
    def elsef(a, b):
      return a + b + 20
    def finallyf(a, b):
      return a + b + 30
    block = ex.tryblock(tryf, (ValueError, proc_ve))
    self.assertRaises(ValueError, block, 0, 4)
    self.assertRaises(ValueError, block, 6, 4)
    self.assertEqual([5, 16, 7, 18, 9], map(lambda v: block(v, 4), xrange(1, 6)))
    block = ex.tryblock(tryf, (ValueError, proc_ve), elsef=elsef)
    self.assertEqual([25, 16, 27, 18, 29], map(lambda v: block(v, 4), xrange(1, 6)))
    block = ex.tryblock(tryf, (ValueError, proc_ve), elsef=elsef, returnbody=True)
    self.assertEqual([5, 16, 7, 18, 9], map(lambda v: block(v, 4), xrange(1, 6)))
    block = ex.tryblock(tryf, (ValueError, proc_ve), finallyf=finallyf)
    self.assertEqual([5, 16, 7, 18, 9], map(lambda v: block(v, 4), xrange(1, 6)))
    block = ex.tryblock(tryf, (ValueError, proc_ve), finallyf=finallyf, returnfinally=True)
    self.assertEqual([35, 36, 37, 38, 39], map(lambda v: block(v, 4), xrange(1, 6)))