[英]Log stderr to file, prefixed with datetime
I do proper logging with the logging
module ( logger.info
, logger.debug
...) and this gets written to a file.我使用
logging
模块( logger.info
、 logger.debug
...)进行适当的日志记录,并将其写入文件。
But in some corner cases (external modules, uncaught exceptions, etc.), I sometimes still have errors written to stderr
.但在某些极端情况下(外部模块、未捕获的异常等),我有时仍然会向
stderr
写入错误。
I log this to a file with:我将其记录到一个文件中:
import sys
sys.stdout, sys.stderr = open("stdout.log", "a+", buffering=1), open("stderr.log", "a+", buffering=1)
print("hello")
1/0
It works, but how to also have the datetime logged before each error?它有效,但如何在每个错误之前记录日期时间?
Note: I'd like to avoid to use logging
for this part, but something more low level.注意:我想避免在这部分使用
logging
,而是使用更底层的东西。
I also want to avoid this solution:我也想避免这种解决方案:
def exc_handler(ex_cls, ex, tb):
with open('mylog.log', 'a') as f:
dt = time.strftime('%Y-%m-%d %H:%M:%S')
f.write(f"{dt}\n")
traceback.print_tb(tb, file=f)
f.write(f"{dt}\n")
sys.excepthook = exc_handler
because some external modules might override this.因为一些外部模块可能会覆盖它。 Is there a low level solution like overriding
sys.stderr.print
?是否有像覆盖
sys.stderr.print
这样的低级解决方案?
You can use the approach described in this cookbook recipe to treat a logger like an output stream, and then redirect sys.std{out,err}
to log stuff written to them.您可以使用本食谱中描述的方法将记录器视为 output stream,然后重定向
sys.std{out,err}
以记录写入它们的内容。 Then, of course, you can configure logging to use whatever format you want, including a date-time prefix, in the normal way using eg %(asctime)s
in the format.然后,当然,您可以配置日志记录以使用您想要的任何格式,包括日期时间前缀,以正常方式使用格式中的
%(asctime)s
。
For the benefit of random readers, if you are simply looking to "log stderr to file" - this is not the answer you are looking for.为了随机读者的利益,如果您只是想“将 stderr 记录到文件”——这不是您要找的答案。 Look at python's
logging
module;查看python的
logging
模块; that is the best approach for most use-cases.这是大多数用例的最佳方法。
With the "disclaimer" part over-with, a lower-level approach to sys.std%s
is very possible, but might be quite tricky.随着“免责声明”部分的结束,对
sys.std%s
的较低级别的方法是很有可能的,但可能非常棘手。 The following example:下面的例子:
sys.stdout, sys.stderr = open(...)
approach from the question itself.sys.stdout, sys.stderr = open(...)
方法。std
writes end with a newline.std
写入都以换行符结尾。#!/usr/bin/env python
import sys
import os
import datetime
import time
import subprocess
from types import MethodType
def get_timestamp_prefix():
return f"{datetime.datetime.now()!s}: "
def stdwriter(self, msg, *args, **kwargs):
tstamp_prefix = get_timestamp_prefix()
if self._last_write_ended_with_newline:
# Add current timestamp if we're starting a new line
self._orig_write(tstamp_prefix)
self._last_write_ended_with_newline = msg.endswith('\n')
if msg.endswith('\n'):
# Avoid associating current timestamp to next error message
self._orig_write(f"\n{tstamp_prefix}".join(msg.split('\n')[:-1]))
self._orig_write("\n")
else:
self._orig_write(f"\n{tstamp_prefix}".join(msg.split('\n')))
def setup_std_files():
sys.stdout, sys.stderr = open("stdout.log", "a+", buffering=1), open("stderr.log", "a+", buffering=1)
for stream_name in ("stdout", "stderr"):
sys_stream = getattr(sys, stream_name)
setattr(sys_stream, "_last_write_ended_with_newline", True)
setattr(sys_stream, "_orig_write", getattr(sys_stream, "write"))
setattr(sys_stream, "write", MethodType(stdwriter, sys_stream))
def print_some_stuff():
print("hello")
print("world")
print("and..,", end=" ")
time.sleep(2.5)
# demonstrating "single" (close enough) timestamp until a newline is encountered
print("whazzzuppp?")
print("this line's timestamp should be ~2.5 seconds ahead of 'and.., whazzzuppp'")
def run_some_failing_stuff():
try:
1/0
except (ZeroDivisionError, subprocess.CalledProcessError) as e:
print(f"Exception handling: {e!r}") # to STDOUT
raise e # to STDERR (only after `finally:`)
else:
print("This should never happen")
finally:
print("Getting outta' here..") # to STDOUT
if __name__ == "__main__":
setup_std_files()
print_some_stuff()
run_some_failing_stuff()
Running this code will output:运行此代码将 output:
$ rm *.log; ./pyerr_division.py ; for f in *.log; do echo "====== $f ====="; cat $f; echo "====== end ====="; done
====== stderr.log =====
2023-01-14 06:59:02.852233: Traceback (most recent call last):
2023-01-14 06:59:02.852386: File "/private/tmp/std_files/./pyerr_division.py", line 63, in <module>
2023-01-14 06:59:02.853152: run_some_failing_stuff()
2023-01-14 06:59:02.853192: File "/private/tmp/std_files/./pyerr_division.py", line 53, in run_some_failing_stuff
2023-01-14 06:59:02.853294: raise e # to STDERR (only after `finally:`)
2023-01-14 06:59:02.853330: File "/private/tmp/std_files/./pyerr_division.py", line 50, in run_some_failing_stuff
2023-01-14 06:59:02.853451: 1/0
2023-01-14 06:59:02.853501: ZeroDivisionError: division by zero
====== end =====
====== stdout.log =====
2023-01-14 06:59:00.346447: hello
2023-01-14 06:59:00.346502: world
2023-01-14 06:59:00.346518: and.., whazzzuppp?
2023-01-14 06:59:02.851982: this line's timestamp should be ~2.5 seconds ahead of 'and.., whazzzuppp'
2023-01-14 06:59:02.852039: Exception handling: ZeroDivisionError('division by zero')
2023-01-14 06:59:02.852077: Getting outta' here..
====== end =====
Child propagation is not explicitly in-scope for this question.子传播不明确在这个问题的范围内。 Still, changing the approach not only allows to collect children's
std%s
to the same files, but also provides for easier debugging.尽管如此,改变方法不仅允许将孩子的
std%s
收集到相同的文件中,而且还提供了更容易的调试。 The idea is to write to the original std%s
if problems occur.如果出现问题,想法是写入原始
std%s
。
The following is based on this answer (thanks user48..2):以下是基于此答案(感谢 user48..2):
def failsafe_stdwriter(self, *args, **kwargs):
try:
self.stdwriter(*args, **kwargs)
except BaseException as be:
try:
self._orig_file.write(*args, **kwargs)
self._orig_file.write(f"\nFAILED WRITING WITH TIMESTAMP: {be!r}\n")
except Exception:
pass
raise be
def setup_std_files():
# sys.stdout, sys.stderr = open("stdout.log", "a+", buffering=1), open("stderr.log", "a+", buffering=1)
for stream_name in ("stdout", "stderr"):
sys_stream = getattr(sys, stream_name)
f = open(f"{stream_name}.log", "a+")
sys_stream_dup = os.dup(sys_stream.fileno())
setattr(sys_stream, "_orig_file", open(sys_stream_dup, "w"))
os.dup2(f.fileno(), sys_stream.fileno())
setattr(sys_stream, "_last_write_ended_with_newline", True)
setattr(sys_stream, "_orig_write", getattr(sys_stream, "write"))
setattr(sys_stream, "write", MethodType(stdwriter, sys_stream))
setattr(sys_stream, "stdwriter", MethodType(stdwriter, sys_stream))
setattr(sys_stream, "write", MethodType(failsafe_stdwriter, sys_stream))
With the above changes, child outputs will also be written to the files.通过上述更改,子输出也将写入文件。 For example, if we replace
1/0
with subprocess.check_call(["ls", "/nosuch-dir"])
, the output will be as follows:例如,如果我们将
1/0
替换为subprocess.check_call(["ls", "/nosuch-dir"])
,则 output 将如下所示:
$ rm *.log; ./pyerr_subprocess.py ; for f in *.log; do echo "====== $f ====="; cat $f; echo "====== end ====="; done
====== stderr.log =====
ls: /nosuch-dir: No such file or directory
2023-01-14 08:22:41.945919: Traceback (most recent call last):
2023-01-14 08:22:41.945954: File "/private/tmp/std_files/./pyerr_subprocess.py", line 80, in <module>
2023-01-14 08:22:41.946193: run_some_failing_stuff()
2023-01-14 08:22:41.946232: File "/private/tmp/std_files/./pyerr_subprocess.py", line 71, in run_some_failing_stuff
2023-01-14 08:22:41.946339: raise e # to STDERR (only after `finally:`)
2023-01-14 08:22:41.946370: File "/private/tmp/std_files/./pyerr_subprocess.py", line 68, in run_some_failing_stuff
2023-01-14 08:22:41.946463: subprocess.check_call(["ls", "/nosuch-dir"])
2023-01-14 08:22:41.946494: File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 373, in check_call
2023-01-14 08:22:41.946740: raise CalledProcessError(retcode, cmd)
2023-01-14 08:22:41.946774: subprocess.CalledProcessError: Command '['ls', '/nosuch-dir']' returned non-zero exit status 1.
====== end =====
====== stdout.log =====
2023-01-14 08:22:39.428945: hello
2023-01-14 08:22:39.428998: world
2023-01-14 08:22:39.429013: and.., whazzzuppp?
2023-01-14 08:22:41.931855: this line's timestamp should be ~2.5 seconds ahead of 'and.., whazzzuppp'
2023-01-14 08:22:41.945638: Exception handling: CalledProcessError(1, ['ls', '/nosuch-dir'])
2023-01-14 08:22:41.945774: Getting outta' here..
====== end =====
Children's output will not have a timestamp with their output (much like ls
's output in the above example), however this not only propagates to children, but also helps debugging (by allowing to fall-back to the original standard outputs).儿童的 output 将没有时间戳与他们的 output(很像上面示例中的
ls
的 output),但这不仅传播给儿童,而且有助于调试(通过允许回退到原始标准输出)。 For example, a simple typo in stdwriter
would be visible, as the output falls-back to the terminal:例如,
stdwriter
中的一个简单拼写错误将是可见的,因为 output 回退到终端:
$ rm *.log; ./pyerr_subprocess.py ; for f in *.log; do echo "====== $f ====="; cat $f; echo "====== end ====="; done
hello
FAILED WRITING WITH TIMESTAMP: NameError("name 'get_tamestamp_prefix' is not defined")
There might be better (and simpler) approaches.可能有更好(和更简单)的方法。 I did not explore the option of subclassing file types at all.
我根本没有探索子类化文件类型的选项。 The one thing I take away from this is - tinkering with error message output methods, makes for an "interesting" debug journey.
我从中得到的一件事是——修改错误消息 output 方法,进行“有趣”的调试之旅。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.