How to make Node JS logs on App Engine standard as nice as Python

Question

I just started running Nodejs on App Engine standard and everything is great except the standard log output is terrible. In Python (on app engine standard) you get really nice looking logs like this:

Where the output is nicely cascaded under the parent when there are multiple lines, the logging level is indicated by the colored icon on the left hand side, etc.

In Node the logs look like this:

As you can see everything get its own line, their is no tree structure and who knows if these are info, warn, or error blocks.

I've tried using Watson and Bunyan with the trace libs as indicated in this SO post: App Engine Node.js: how to link app logs and requests logs

but it doesn't seem to work. It would be great for Google to make this work by default as they do with Python or Java, barring that it would be nice to get a working demo, I've also tried to the node lib for logging that is part of the google apis collection to no avail.

Can anyone point me to an example of how to fix the logs so I can match what exists already in Python for app engine standard?

Thanks for the help!

UPDATE -- Still not working

So after deploying the hello-world sample and adding one logger to the app get method now looks like this:

app.get('/', (req, res) => {
    var data_block = {
    "glossary": {
        "title": "example glossary",
        "GlossDiv": {
            "title": "S",
            "GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
                    "SortAs": "SGML",
                    "GlossTerm": "Standard Generalized Markup Language",
                    "Acronym": "SGML",
                    "Abbrev": "ISO 8879:1986",
                    "GlossDef": {
                        "para": "A meta-markup language, used to create markup languages such as DocBook.",
                        "GlossSeeAlso": ["GML", "XML"]
                    },
                    "GlossSee": "markup"
                }
            }
        }
    }
}

    console.log("This is a really big log request that would normally not work very well:", data_block)
  res.status(200).send('Hello, world!').end();
});

The output looks like this:

You can see that the logs aren't like the Python logs, any idea what I am doing wrong?

Answer 1

You can use the Bunyan client library to stream logs to Stackdriver. Here's an example based on the documentation and the code sample you provided.

const loggingBunyan = new LoggingBunyan();
const logger = bunyan.createLogger({
    name: 'my-service',
    level: 'info',
    streams: [
            {stream: process.stdout},
            loggingBunyan.stream(),
        ],
});

app.get('/', (req, res) => {
  var data_block = {
    "glossary": {
        "title": "example glossary",
        "GlossDiv": {
            "title": "S",
            "GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
                    "SortAs": "SGML",
                    "GlossTerm": "Standard Generalized Markup Language",
                    "Acronym": "SGML",
                    "Abbrev": "ISO 8879:1986",
                    "GlossDef": {
                        "para": "A meta-markup language, used to create markup languages such as DocBook.",
                        "GlossSeeAlso": ["GML", "XML"]
                    },
                    "GlossSee": "markup"
                }
            }
        }
    }
  }
  res.status(200).send('Hello, world!').end();
  logger.info(data_block);
});

This will log the entire JSON block as an individual message, however the documentation also describes how to nest application logs under request logs, if that's what you want to do.

Answer 2

You can fix it by using console.log(JSON.stringify(event)); instead of console.log(event); to log your JavaScript objects.

Answer 3

If you come from App Engine Standard Pyton (2) the logging experience is quite frustrating. The documentation on Google Cloud Services is a mess, scattered and often outdated. I found no example for best practices of high preformance logging on google App Engine node.js standard environment. So how to get decent, aggregated logging?

I think part of the problem is the fact that for Python the Google Infrastructure calls your code on a per request basis (somewhat an early implementation of cloud functions) while with node.js you have a long running process which is contacted by the infrastructure via HTTP/TCP/IP on localhost. So the GAE infrastructure can not thell easily which output belongs to which request.

Google suggests:

To write log entries, we recommend that you integrate the Bunyan or Winston plugins with Cloud Logging. ( source )

But using an in-process Cloud Logging client is generally an problematic approach for the async / single process server approach offered by node.js / Express. My Testing indicates that per logger call you get one outbound HTTP request to Stackdriver / Clound Logging. Batching seems to be possible but would been a pain to implement in failure situations.

Logging to stdout / stderr and transporting logs from there to the log viewer in a separate process (provided by App Engine infrastructure) provides much better decoupeling. And it should be cheaper because semingly you dont pay for the cycles spend there (?). This shoud be possible, as stated by Google:

you can send simple text strings to stdout and stderr. The strings will appear as messages in the Logs Viewer, […] If you want to filter these strings in the Logs Viewer by severity level, you need to format them as structured data. For more information, see Structured logging. If you want to correlate entries in the app log with the request log, your structured app log entries need to contain the request's trace identifier. […] You can use the same technique in your App Engine apps. ( source )

To get structured logging output (basically lines of JSON) something like Pino is the right aproach. You need some setup to map Pino field names to the ones used by Stackdriver / Google Cloud Logging.

Interesting doumentation for field Names at Google is somewhat scattered:

• severity
• httpRequest
• operation

Some mapping has to be done. Setting up Pino for Google / Stackdriver looks like this:

// https://github.com/pinojs/pino/blob/master/docs/api.md#level-string
// https://cloud.google.com/logging/docs/agent/configuration#special-fields
function _levelToSeverity (level) {
  if (level < 30) { return 'DEBUG' }
  if (level <= 30) { return 'INFO' }
  if (level <= 39) { return 'NOTICE' }
  if (level <= 49) { return 'WARNING' }
  if (level <= 59) { return 'ERROR' }
  if (level <= 69) { return 'CRITICAL' } // also fatal
  if (level <= 79) { return 'ALERT' }
  if (level <= 99) { return 'EMERGENCY' }
  return 'DEFAULT'
}
const logger = pino({
  // Default Level: `info` or taken from the environment Variable LOG_LEVEL
  // levels below this get supressed so set this to `debug` or `trace`
  // if you want to be able to reveal details in the App Engine logs
  level: process.env.LOG_LEVEL || 'info',
  formatters: { // Here the Mapping is happening
    level (label, number) {
      return { level: number, severity: _levelToSeverity(number) }
    }
  },
  messageKey: 'message' // AppEngine / StackDriver want's this key
  // Enable Pretty Printing on the development Server
  prettyPrint: (process.env.NODE_ENV || 'development') === 'development',
})

This enables us to write out logs in a format which can be digested by the Google Log Viewer and allows us to 'drill down' in the logs.

Next thig is to enable logging per HTTP request. pino-http uses the logger defined above and can be wired into the Google machinery to use Googles Request-IDs:

const httpLogger = require('pino-http')({
  logger: logger,
  genReqId: function (req) {
    // get a Google rrovided request ID to correlate entries
    return (
      req.headers['x-appengine-request-log-id'] ||
      req.headers['x-cloud-trace-context'] ||
      req.id
    )
  },
  useLevel: 'debug'
})

This adds a log Instance to every Express-Request and can be used in your route hadlers as req.log.info('bar') etc. You can also add structured fields to your logging like req.log.info({user: 'mike', action: 'create'}, 'foo') . These fields can be queried in the log viewer.

I usually do not use pino-http because it adds to much clutter. Instead I generate the requestid by myself.

If you want to correlate entries in the app log with the request log, your structured app log entries need to contain the request's trace identifier. You can extract the trace identifier from the X-Cloud-Trace-Context request header. In your structured log entry, write the ID to a field named logging.googleapis.com/trace. For more information about the X-Cloud-Trace-Context header (X-Cloud-Trace-Context). ( source )

This can be done by a Express-Middleware:

const gaeTraceMiddleware = (req, res, next) => {
  // Add log correlation to nest all log messages beneath request log in Log Viewer.
  const traceHeader = req.header('X-Cloud-Trace-Context')
  if (traceHeader && process.env.GOOGLE_CLOUD_PROJECT) {
    const [trace] = traceHeader.split('/')
    req.log = res.log = req.log.child({
      'logging.googleapis.com/trace': `projects/${process.env.GOOGLE_CLOUD_PROJECT}/traces/${trace}`,
      'appengine.googleapis.com/request_id': req.header('x-appengine-request-log-id')
    })
  }
  next()
}

Now the Express instance wired up to gaeTraceMiddleware can use res.log to output stuff for Google Log Viewer. There it will appear nearly as neat as Pytohn logs:

const app = express()
app.use(httpLogger)
app.use(gaeTraceMiddleware)

// App Engine Instance Start
app.get('/_ah/warmup', (req, res) => {
  res.log.info('Warmup')
  res.send('HOT')
})

But you still will be missing per request aggregation but still the logging experience get's much better with this.