写入python日志的最快方法

Question

I am using the gevent loop in uWSGI and I write to a redis queue. I get about 3.5 qps. On occasion, there will be an issue with the redis connection so....if fail, then write to a file where I will have a separate process do cleanup later. Because my app very latency aware, what is the fastest way to dump to disk in python? Will python logging suffice?

Answer 1

If latency is a crucial factor for your app, undefinitely writing to disk could make things really bad.

If you want to survive a reboot of your server while redis is still down i see no other solutions than writing to disk, otherwise you may want to try with a ramdisk.

Are you sure having a second server with a second instance of redis would not be a better choice ?

Regarding logging, i would simply use low-level I/O functions as they have less overhead (even if we are talking of very few machine cycles)

Answer 2

Appending to a file on disk is fast.

:~$ time echo "this happened" >> test.file

real    0m0.000s
user    0m0.000s
sys     0m0.000s

Appending to a file with Python seems to be about the same order of magnitude as bash. The logging module does seem to add a little bit of overhead:

import logging
import time

logging.basicConfig(filename='output_from_logging')
counter = 0

while counter < 3:
    start = time.time()
    with open('test_python_append', 'a') as f:
        f.write('something happened')
    counter += 1
    print 'file append took ', time.time() - start

counter = 0
while counter < 3:
    start = time.time()
    logging.warning('something happened')
    counter += 1
    print 'logging append took ', time.time() - start

Which gives us this output:

file append took  0.000263929367065
file append took  6.79492950439e-05
file append took  5.41210174561e-05
logging append took  0.000214815139771
logging append took  0.0001220703125
logging append took  0.00010085105896

But in the grand scheme of things I doubt that this operation will be a very costly part of your code base and is probably not worth worrying about. If you are worried about latency then you should profile your code python -m cProfile code_to_test.py . That will tell you how long each of your functions is taking and where your application is spending time. I seriously doubt it will mainly be in logging errors.