How to read and delete first n lines from file in Python - Elegant Solution[2]

Question

Originally posted here: How to read and delete first n lines from file in Python - Elegant Solution

I have a pretty big file ~ 1MB in size and I would like to be able to read first N lines, save them into a list ( newlist ) for later use, and then delete them.

My original code was:

import os

n = 3 #the number of line to be read and deleted

with open("bigFile.txt") as f:
    mylist = f.read().splitlines()

newlist = mylist[:n]
os.remove("bigFile.txt")

thefile = open('bigFile.txt', 'w')

del mylist[:n]

for item in mylist:
  thefile.write("%s\n" % item)

Based on Jean-François Fabre code that was posted and later deleted here I am able to run the following code:

import shutil

n = 3

with open("bigFile.txt") as f, open("bigFile2.txt", "w") as f2:
    for _ in range(n):
        next(f)
    f2.writelines(f)

This works great for deleting the first n lines and "updating" the bigFile.txt but when I try to store the first n values into a list so I can later use them like this:

with open("bigFile.txt") as f, open("bigFile2.txt", "w") as f2:
    mylist = f.read().splitlines()
    newlist = mylist[:n]
    for _ in range(n):
        next(f)
    f2.writelines(f)

I get an "StopIteration" error

Answer 1

In your sample code you are reading the entire file to find the first n lines:

# this consumes the entire file
mylist = f.read().splitlines()

This leaves nothing left to read for the subsequent code. Instead simply do:

with open("bigFile.txt") as f, open("bigFile2.txt", "w") as f2:
    # read the first n lines into newlist
    newlist = [f.readline() for _ in range(n)]
    f2.writelines(f)

Answer 2

I would proceed as follows:

n = 3
yourlist = []
with open("bigFile.txt") as f, open("bigFile2.txt", "w") as f2:
    i=0
    for line in f:
        i += 1
        if i<n:
            yourlist.append(line)
        else:
            f2.write(f)