Raw Custom Logs in Google App Engine Flexible Environment

Question

I've used the following code in a Flexible (VM) Environment on Google App Engine to separate custom logs with a specific formatting requirement from other application logs:

import logging as std_logging

std_logging.basicConfig()

custom_formatter = std_logging.Formatter('%(created)f\t%(message)s')
custom_handler = std_logging.FileHandler('/var/log/app_engine/custom_logs/custom.log')
custom_handler.setFormatter(custom_formatter)
custom_logging = std_logging.getLogger('custom')
custom_logging.addHandler(custom_handler)

In a normal Python environment, these would write to the log file as plain-text lines in the format specified.

However, after dumping the logs produced from App Engine to Cloud Storage, I noticed that App Engine has wrapped each log with other information.

Eg (formatted for clarity)

{
    "insertId":"vsdacv1235jj1",
    "log":"appengine.googleapis.com/custom.var.log.app_engine.app.custom_logs.custom.log",
    "metadata":{
        "labels":{
            "appengine.googleapis.com/module_id":"default",
            "appengine.googleapis.com/version_id":"1",
            "compute.googleapis.com/resource_id":"1234256789901203",
            "compute.googleapis.com/resource_name":"bbq23asd123",
            "compute.googleapis.com/resource_type":"instance"
        },
        "projectId":"my-project",
        "serviceName":"appengine.googleapis.com",
        "timestamp":"2016-06-24T20:16:15Z",
        "zone":"us-central1-f"
    },
    "textPayload":"1466799374933\tthis is my custom message"
}

The value of the textPayload field is the actual log I produced, but is wrapped by App Engine.

Is there a way to prevent this behaviour? It is undesirable to re-process these logs in order to format them correctly.

Answer 1

The extra information is actually quite handy, especially in an appengine environment where filtering on module/service, version, or instance_id can really help you investigate issues.

If reprocessing the logs is your problem, I would suggest turing on the export to BigQuery functionality . Logs will be constantly streamed to BigQuery and immediately queriable. The nice thing here is you could query for just the text_payload property and easily export just this result set to csv, text, json, etc.

As far as exporting just lines of text from logs, I do not believe this is possible.