Tornado - how does application and request logging work?

Question

I'm using tornado 3.2.1, python 2.7.3, supervisord 3.0 on AWS EC2 using Amazon Linux AMI.

I'm trying to get request level logging (eg "200 POST / (127.0.0.1) 0.75ms" (this is the definition of request level logging, right?)) to write to a log file and exception stack traces to log to another log file. I see from here that there are three potential logs available as part of Tornado, and ideally, i'd also like output reported from that general log writing to another file. I've read its better to have supervisord (or something other than Tornado) handle the writing of the log files.

My supervisord.conf looks like this:

[unix_http_server]
file=/var/lib/supervisor/supervisor.sock   ; (the path to the socket file)

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/lib/supervisor/supervisor.sock ; use a unix:// URL  for a unix socket

[include]
files=supervisord.d/*.ini

This is my app supervisord ini file:

[program:myapp]
process_name=myapp%(process_num)s
directory=/opt/me/venvs/myapp
command=/opt/me/venvs/myapp/bin/python /opt/me/venvs/myapp/bin/run_myapp.py --port=%(process_num)s --logging=DEBUG
startsecs=2
user=me
stdout_logfile=/var/log/myapp/access-%(process_num)s.log
stderr_logfile=/var/log/myapp/error-%(process_num)s.log
numprocs=2
numprocs_start=14000

and I have this code snippet at the bottom of run_myapp.py

if __name__ == "__main__":
    tornado.options.define("port", default=7777, help="run on the given port", type=int)
    tornado.options.parse_command_line()
    http_server = tornado.httpserver.HTTPServer(Application())
    http_server.listen(tornado.options.options.port)
    tornado.ioloop.IOLoop.instance().start()

and what seems to happen is that request level info, stack traces, and uses of logging.info('doing something...') all get recorded to the file designated by supervisord stderr_logfile variable. Nothing gets recorded to the stdout_logfile. The only time I can get anything recording to the stdout_logfile is if I actually put print statements in the code and use sys.stdout.flush() to flush the buffer (or start python with -u flag).

What is the correct way to setup logging of requests to the stdout_logfile, stacktraces and other application level output to the stderr_logfile? I suppose the second part of the question is already answered since it's already doing that, but the first part is so far unsolved.

Answer 1

Tornado's built-in log configuration in parse_command_line cannot help you in separating your log streams - you'll have to set tornado.options.options.logging=None and manage your own configuration in the python logging modules directly.

Supervisord can only handle two log files (stdout and stderr), so you cannot both let supervisord handle your log files and have fine-grained separation, but you can have two streams. Configure the tornado.access logger to write to stdout with propagate=False, and the root logger to write to stderr. Then you can give the two streams different output files in supervisord.