Most efficient way to “nibble” the first line of text from a text document then resave it in python

Question

I have a text document that I would like to repeatedly remove the first line of text from every 30 seconds or so.

I have already written (or more accurately copied) the code for the python resettable timer object that allows a function to be called every 30 seconds in a non blocking way if not asked to reset or cancel.

Resettable timer in python repeats until cancelled

(If someone could check the way I implemented the repeat in that is ok, because my python sometimes crashes while running that, would be appreciated :))

I now want to write my function to load a text file and perhaps copy all but the first line and then rewrite it to the same text file. I can do this, this way I think... but is it the most efficient ?

def removeLine():

    with open(path, 'rU') as file:
        lines = deque(file)
        try:
            print lines.popleft()
        except IndexError:
            print "Nothing to pop?"
    with open(path, 'w') as file:
        file.writelines(lines)  

This works, but is it the best way to do it ?

Answer 1

I'd use the fileinput module with inplace=True :

import fileinput

def removeLine():
    inputfile = fileinput.input(path, inplace=True, mode='rU')
    next(inputfile, None)  # skip a line *if present*
    for line in inputfile:
        print line,  # write out again, but without an extra newline
    inputfile.close()

inplace=True causes sys.stdout to be redirected to the open file, so we can simply 'print' the lines.

The next() call is used to skip the first line; giving it a default None suppresses the StopIteration exception for an empty file.

This makes rewriting a large file more efficient as you only need to keep the fileinput readlines buffer in memory.

I don't think a deque is needed at all, even for your solution; just use next() there too, then use list() to catch the remaining lines:

def removeLine():
    with open(path, 'rU') as file:
        next(file, None)  # skip a line *if present*
        lines = list(file)
    with open(path, 'w') as file:
        file.writelines(lines)  

but this requires you to read all of the file in memory; don't do that with large files.