Best practices for logging in Python in shared libraries

Question

In a nutshell, I write ETL pipelines. They are usually described in high-level scripts. In them, I use different internal libraries (we manage them) that provide utility functions, tooling or internal data structure.

What are the common best practices about logging when dealing with multiple packages import from different repositories?

My questions are:

1) Should I put logs in libraries? Or only in top-level scripts?

On one hand, It could be useful to display some information in some library functions/classes. On the other hand, it imposes the library client usage of a particular logger.

I checked a few open-source projects and it seems that there are no logs at all.

2) If we indeed put logs in all shared libraries, what is the best practices in Python to pass a unique logger to everything?

I want my logging format and strategy to be consistent in each library call as everything is run as part "as a whole". Should I init my logger in the main script and pass the same logger in every object I create? It seems redundant to me. I saw another pattern where all classes that need logging would inherit from a logging class. It seems to me that it might overkill and complicates the overall architecture.

I read in another stackoverflow that Actually every logger is a child of the parent's package logger . How to apply that when the packages come different repositories?

thanks

Answer 1

Add a logger with no handlers (or with just the null handler ) to the library and do all the internal logging with that. Give it a name that is related to the library. When you do that any app that uses the lib can get the logger and add a handler to access the logs as needed.

An example would be the requests library which does something similar to that.

import logging
import requests
r = logging.getLogger('requests')
r.addHandler(logging.StreamHandler())
r.setLevel(logging.DEBUG)
requests.get('http://stackoverflow.com')

will print

Starting new HTTP connection (1): stackoverflow.com
http://stackoverflow.com:80 "GET / HTTP/1.1" 301 143
Starting new HTTPS connection (1): stackoverflow.com
https://stackoverflow.com:443 "GET / HTTP/1.1" 200 23886