How to decode the persisted log files from App Engine's Python Dev_server

Question

Using Google App Engine's local dev_server, one of the published options is to save the log files to disk.

"If you wish to persist the logs from the development server to disk at a location of your own >choosing, supply the desired path and filename to the --logs_path command line option as follows:

dev_appserver.py --logs_path=your-path/your-logfile-name your-app-directory"

This works except the files are in a binary format I can't decode.

Google published some sample code but this is intended for an app to read the production logs.

Is there a command line utlity or python script that can decode the log files stored by the dev_server? This was submitted as a feature request in 2011.

I can see the logs in the PyDev console as they are produced, but once the current execution is restarted, those logs are gone from that console.

There are some suggestions to pipe the logfile unix style, but this does not help with the logs already stored.

Answer 1

The logfile created is a SQLite database. Use the sqlite3 module to access it.

The schema used is:

CREATE TABLE AppLogs (
  id INTEGER NOT NULL PRIMARY KEY,
  request_id INTEGER NOT NULL,
  timestamp INTEGER NOT NULL,
  level INTEGER NOT NULL,
  message TEXT NOT NULL,
  FOREIGN KEY(request_id) REFERENCES RequestLogs(id)
);
CREATE TABLE RequestLogs (
  id INTEGER NOT NULL PRIMARY KEY,
  user_request_id TEXT NOT NULL,
  app_id TEXT NOT NULL,
  version_id TEXT NOT NULL,
  module TEXT NOT NULL,
  ip TEXT NOT NULL,
  nickname TEXT NOT NULL,
  start_time INTEGER NOT NULL,
  end_time INTEGER DEFAULT 0 NOT NULL,
  method TEXT NOT NULL,
  resource TEXT NOT NULL,
  http_version TEXT NOT NULL,
  status INTEGER DEFAULT 0 NOT NULL,
  response_size INTEGER DEFAULT 0 NOT NULL,
  user_agent TEXT NOT NULL,
  url_map_entry TEXT DEFAULT '' NOT NULL,
  host TEXT NOT NULL,
  task_queue_name TEXT DEFAULT '' NOT NULL,
  task_name TEXT DEFAULT '' NOT NULL,
  latency INTEGER DEFAULT 0 NOT NULL,
  mcycles INTEGER DEFAULT 0 NOT NULL,
  finished INTEGER DEFAULT 0 NOT NULL
);

RequestLogs contains HTTP requests (from a browser or a task queue); AppLogs contains any logging entries recorded during a specific request.