C Remove the first line from a text file without rewriting file

Question

I've got a service which runs all the time and also keeps a log file. It basically adds new lines to the log file every few seconds. I'm written a small file which reads these lines and then parses them to various actions. The question I have is how can I delete the lines which I have already parsed from the log file without disrupting the writing of the log file by the service?

Usually when I need to delete a line in a file then I open the original one and a temporary one and then I just write all the lines to the temp file except the original which I want to delete. Obviously this method will not word here.

So how do I go about deleting them ?

Answer 1

In most commonly used file systems you can't delete a line from the beginning of a file without rewriting the entire file. I'd suggest instead of one large file, use lots of small files and rotate them for example once per day. The old files are deleted when you no longer need them.

Answer 2

Can't be done, unfortunately, without rewriting the file, either in-place or as a separate file.

One thing you may want to look at is to maintain a pointer in another file, specifying the position of the first unprocessed line.

Then your process simply opens the file and seeks to that location, processes some lines, then updates the pointer.

You'll still need to roll over the files at some point lest they continue to grow forever.

Answer 3

I'm not sure, but I'm thinking in this way: New Line is a char, so you must delete chars for that line + New Line char By the way, "moving" all characters back (to overwrite the old line), is like copying each character in a different position, and removing them from their old position

So no, I don't think you can just delete a line, you should rewrite all the file.

Answer 4

You can't, that just isn't how files work.

It sounds like you need some sort of message logging service / library that your program could connect to in order to log messages, which could then hide the underlying details of file opening / closing etc.

Answer 5

If each log line has a unique identifier (or even just line number), you could simply store in your log-parsing the identifier until which you got parsing. That way you don't have to change anything in the log file.

If the log file then starts to get too big, you could switch to a new one each day (for example).