使用python在文件中跳过最后5行

Question

I wanted to remove last few lines in a file using python. The file is huge in size,so to remove first few line I'm using the following code

import sys
with open(sys.argv[1],"rb") as f:
    for _ in range(6):#skip first 6 lines
        next(f)
    for line in f:
        print line

Answer 1

Here's a generalized generator for truncating any iterable:

from collections import deque

def truncate(iterable, num):
    buffer = deque(maxlen=num)
    iterator = iter(iterable)

    # Initialize buffer
    for n in range(num):
        buffer.append(next(iterator))

    for item in iterator:
        yield buffer.popleft()
        buffer.append(item)

truncated_range20 = truncate(range(20), 5)

print(list(truncated_range20))
# [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]

Using truncate , you can do this:

from __future__ import print_function

import sys

from itertools import islice


filepath = sys.argv[1]

with open(filepath, 'rb') as f:
    for line in truncate(islice(f, 6, None), 5):
        print(line, end='')

Answer 2

If every line has a different length, and you can't predict when to stop with the file size, your python script has no way to know.

So you need to do some buffering. The easier way is to buffer the whole file, split everything in lines, and then remove that last 5, but you seem to say that you can't, because the file is huge.

So why not keep only the 5 last lines in memory?

import sys

with open(sys.argv[1],"rb") as f:
    # Skip 6 lines
    for _ in range(6):
        next(f)

    # Create a list that will contain at most 5 lines.
    # Using a list is not super efficient here (a Queue would be better), but it's only 5 items so...
    last_lines = []
    for line in f:
        # if the buffer is full, print the first one and remove it from the list.
        if len(last_lines) == 5:
            print last_lines.pop(0)

        # append current line to the list.
        last_lines.append(line)

    # when we reach this comment, the last 5 lines will remain on the list.
    # so you can just drop them.

As a side note, i suppose that you explicitely said that you want to use python, because you want to replace the "print line" with something else later, or do some additional processing.

If you are not, use your operating system "head" and "tail" commands (i have no idea how they are named on windows though), which will be much more faster (because they use better data structures, read and process big blocks at once, scan the file from the end, are not coded using python, etc).

Answer 3

The following works nicely, and would be suitable for very large files.

It opens the file for updating, skips to almost the end and reads the remaining portion as lines. It then moves the file pointer back to where it started reading from. It then writes back all but the last 5 lines to the file, and truncates the remaining part of the file:

import os 

back_up = 5 * 200       # Go back from the end more than 5 lines worth

with open("foo.txt", "r+") as f:
    f.seek(-back_up, os.SEEK_END)
    lines = f.readlines()[:-5]
    f.seek(-back_up, os.SEEK_END)
    f.write("".join(lines))
    f.truncate()

You must decide how long you feel each line could roughly be. It does not need to be an exact value, just enough to ensure you have the last lines.

For example if your lines are very long, you could back_up a much larger value, eg 10 * 10000 to be on the safe side. This would avoid you having to process the whole of your large file.