阅读不断增长的文件

Question

I ran the command PROGRAM 2> log.txt ,

and it will write the output to log.txt continuously.

and the output will in the same line, because every the output only has \\r in the end.

I'm trying to read the log.txt in that way, with truncate to clear the data I've read.

    read_in_file = open(in_file, 'r')
    records = []
    for new_item in self.__get_matched_line(read_in_file):
        records.append(new_item)

    read_in_file.truncate()
    read_in_file.close()
    return records

But it has some problem, I still get the have-read data sometime.

The program may run for weeks.and it outputs every 0.5 seconds

Answer 1

Your PROGRAM will have an open file handle on the output file; therefore even if you truncate the file from the outside, the file handle will write to its old position, no matter how your file had been truncated. The result will be a file of the original length before the truncation (the gap is filled with zeros, maybe read about sparse files for more details if you choose this path). You can circumvent that by closing the file handle and opening it anew for each line. This is best done within PROGRAM of course. If you cannot change that, you can read line-by-line (delimited by '\\r' ) and append whole lines to your log file:

PROGRAM |& while IFS= read -e -d $'\r' line; do echo "$line" >> log.txt; done

In case you really only want stderr to be logged this way, use this pattern:

(PROGRAM 2>&1 1>&3 | while IFS= read -e -d $'\r' line; do echo "$line" >> log.txt; done) 3>&1

This way, a truncate on the log file will have the desired effect.

But consider that using truncate will only remove stuff from the end of the file while keeping the beginning. There is no way to keep the end while removing the beginning if you use just one file. So using truncate to remove the parts you already read only makes sense if you truncate the whole file (set it to 0 bytes).

So you might want to consider log rotation . That's a well-defined process which may have advantages over your approach (and be it only that there are facilities which already do it for you).