Google StackDriver correlating logs with parent request python 3

Question

In python 2.7, the app engine sdk was doing the work in the background to nest all logs with the parent request to have a correlation in Google StackDriver.

As of the transition to python 3, it is through the usage of google cloud logging or structured logging, and from all the different references I could found, it's important to have the same trace id in the 'sub' logs for stack driver to make a match with the 'request' log.

And still as you can see below, it still appear as different logs.

For context, I even tried this on an empty django project deployed on app engine.

Got the same result, even when following the example in the documentation: https://cloud.google.com/run/docs/logging#writing_structured_logs

Trying to log to the stdout is giving the same result.

Edit:

After the initial request, all other request will be nested under the initial request when using the stdout.

But, the highest severity of the 'child' logs is not taken by the 'parent' log, therefore the filters won't pick up the actual log. See below:

Answer 1

Thanks for the question!

It looks like you're logging the trace correctly, but your logName indicates that you're not using a stdout or stderr. If you use one of these for your logs, they will correlate properly, like this:

StackDriver Logs Screenshot

You can see that the logName ends with stdout. An stdout or stderr will correlate. You can create this as shown here in the tutorial:

# Build structured log messages as an object.
global_log_fields = {}

# Add log correlation to nest all log messages
# beneath request log in Log Viewer.
trace_header = request.headers.get('X-Cloud-Trace-Context')

if trace_header and PROJECT:
    trace = trace_header.split('/')
    global_log_fields['logging.googleapis.com/trace'] = (
        f"projects/{PROJECT}/traces/{trace[0]}")

# Complete a structured log entry.
entry = dict(severity='NOTICE',
             message='This is the default display field.',
             # Log viewer accesses 'component' as jsonPayload.component'.
             component='arbitrary-property',
             **global_log_fields)

print(json.dumps(entry))

EDIT:

To filter out the stdout and only see the Request logs in the stackdriver UI, you can de-select the stdout from the filter. Logs Filter

For a sample using the python client API, please see this article and attached sample Flask app. Combining correlated Log Lines in Google Stackdriver

Answer 2

I was able to achieve this kind of logging structure on Google Cloud Logging Console:

I was using the Django Framework. I wrote Django middleware which integrates Google Cloud Logging API.

"Trace" needs to be added to every log object which points to its parent log object.

please check manage nesting of logs in Google Stackdriver with Django

Please check django-google-stackdriver-nested-logging log_middleware.py source on Github.