Split file after X lines at blank line

Question

I need to split large textfiles into smaller chunks whereby the textfiles contain data that needs to stay together. Each related chunk of data is separated from the next by a newline, like so:

Some Data belonnging to chunk 1
Some Data belonnging to chunk 1
Some Data belonnging to chunk 1

More Data, belonnging to chunk 2
More Data, belonnging to chunk 2
More Data, belonnging to chunk 2

How could I define a number of lines after which, at the next blank line to maintain the data chunks, the file is split? I'd like to use Python for this but I can't figure out to use a split function after X lines.

Answer 1

from itertools import groupby

with open(myfile, 'r') as f:
    chunks = [[x.strip() for x in v] for k, v in 
              groupby(f, lambda x: x.strip()) if k]

Answer 2

If you want to write new chunk1.txt ... chunkN.txt for each chunk, you could do this in a manner like this:

def chunk_file(name, lines_per_chunk, chunks_per_file):

    def write_chunk(chunk_no, chunk):
        with open("chunk{}.txt".format(chunk_no), "w") as outfile:
            outfile.write("".join(i for i in chunk))

    count, chunk_no, chunk_count, chunk = 1, 1, 0, []
    with open(name, "r") as f:
        for row in f:
            if count > lines_per_chunk and row == "\n":
                chunk_count += 1
                count = 1
                chunk.append("\n")
                if chunk_count == chunks_per_file:
                    write_chunk(chunk_no, chunk)
                    chunk = []
                    chunk_count = 0
                    chunk_no += 1
            else:
                count += 1
                chunk.append(row)
    if chunk:
        write_chunk(chunk_no, chunk)

chunk_file("test.txt", 3, 1)

You have to specify the lines which belong to a chunk, after which a newline is anticipated.

Say you want to chunk this file:

Some Data belonnging to chunk 1

Some Data belonnging to chunk 1
Some Data belonnging to chunk 1
Some Data belonnging to chunk 1
Some Data belonnging to chunk 1
Some Data belonnging to chunk 1

More Data, belonnging to chunk 2
More Data, belonnging to chunk 2
More Data, belonnging to chunk 2

The first chunk differs strongly in line count from the second chunk. (7 lines vs 3 lines)

The output for this example would be chunk1.txt :

Some Data belonnging to chunk 1

Some Data belonnging to chunk 1
Some Data belonnging to chunk 1
Some Data belonnging to chunk 1
Some Data belonnging to chunk 1
Some Data belonnging to chunk 1

And chunk2.txt :

More Data, belonnging to chunk 2
More Data, belonnging to chunk 2
More Data, belonnging to chunk 2

This approach assumes that lines_per_chunk is a minimum chunk size , so it works even if the chunks have different line counts. We are only looking for a blank line to end the chunk, when the minimum chunk size is reached. In the above example it is no problem, that there is a blank line on line 2, since the minimum chunk size is not reached yet. If a blank line occurs on line 4 and the chunk data continues afterwards, there would be a problem, since the criterion specified (line numbers and blank lines) could not identify chunks alone.