每50,000行分割并在python中写入文件

Question

I read files which contains 75,151 lines. I want to split every 50,000 lines. So I made 2 files, one files have 50,000 lines and the other file has 25,151 lines.

I made code like this (INSERT_NUMBER : 50,000)

 for index, data in enumerate(lines):
   if ((index % INSERT_NUMBER) == 0 and index != 0) or (index == (lines- 1)) : 
           made file ....

which is the better way to split every 50,000 lines and make new files ?

Answer 1

Here's one way using itertools.groupby() :

from itertools import groupby

out_filename = '/tmp/f{}.txt'
lines_per_file = 50000

with open('infile.txt') as infile:
    for file_number, lines in groupby(enumerate(infile), key=lambda x: x[0] // lines_per_file):
        with open(out_filename.format(file_number), 'w') as outfile:
            outfile.writelines(line for line_number, line in lines)

So the trick here is to use the line number of each line to group it into chunks using integer division, and then to use the grouping key as a counter for the output file name.

Is it better than what you already have? It's a bit more complicated to read the code, but it doesn't need to deal with those annoying edge cases when you try to group using modulo arithmetic.