python WatchedFileHandler still writing to old file after rotation

Question

I've been using WatchedFileHandler as my python logging file handler, so that I can rotate my logs with logrotate (on ubuntu 14.04), which you know is what the docs say its for. My logrotate config files looks like

/path_to_logs/*.log {
        daily
        rotate 365
        size 10M
        compress
        delaycompress
        missingok
        notifempty
        su root root
}

Everything seemed to be working just fine. I'm using logstash to ship my logs to my elasticsearch cluster and everything is great. I added a second log file for my debug logs which gets rotated but is not watched by logstash. I noticed that when that file is rotated, python just keeps writing to /path_to_debug_logs/*.log.1 and never starts writting to the new file. If I manually tail /path_to_debug_logs/*.log.1 , it switches over instantly and starts writing to /path_to_debug_logs/*.log .

This seems REALLY weird to me.

I believe what is happening is that logstash is always tailing my non-debug logs, which some how triggers the switch over to the new file after logrotate is called. If logrotate is called twice without a switch over, the log.1 file gets moved and compressed to log.2.gz, which python can no longer log to and logs are lost.

Clearly there are a bunch of hacky solutions to this (such as a cronjob that tails all my logs every now and then), but I feel like I must be doing something wrong.

I'm using WatchedFileHandler and logrotate instead of RotatingFileHandler for a number of reasons, but mainly because it will nicely compress my logs for me after rotation.

UPDATE:

I tried the horrible hack of adding a manual tail to the end of my log rotation config script.

sharedscripts
postrotate
    /usr/bin/tail -n 1 path_to_logs/*.log.1
endscript

Sure enough this works most of the time, but randomly fails sometimes for no clear reason, so isn't a solution. I've also tried a number of less hacky solutions where I've modified the way WatchFileHandler checks if the file has changed, but no luck.

I'm fairly sure the root of my problem is that the logs are stored on a network drive, which is somehow confusing the file system.

I'm moving my rotation to python with RotatingFileHandler , but if anyone knows the proper way to handle this I'd love to know.

Answer 1

Use copytruncate option of logrotate. From docs

copytruncate

Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one, It can be used when some program can not be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the create option will have no effect, as the old log file stays in place.

Answer 2

WatchedFileHandler does a rollover when a device and/or inode change is detected in the log file just before writing to it. Perhaps the file which isn't being watched by logstash doesn't see a change in its device/inode? That would explain why the handler keeps on writing to it.