Creating Multiple files from a data source in Python

Question

I have a data source I'm working with in Python. I'd like to save that data to a files such that once a threshold is hit (ie: 1K, 1M) the file is closed and a new file is automatically opened to save the data.

ie:

<file handler with buffer 200>
file.write('a'*1000)

The line above would generate 5 files based on the data. Is there a pre-built python library that will handle this, or do I need to write one myself?

Answer 1

If a logger framework is too much, you can do it yourself -- shouldn't need more than a dozen lines of code or so. The easiest way to get the size of your file is by calling the tell() method of your open file descriptor.

You could also keep track of the bytes being output, but this requires additional logic if your program sometimes appends to a pre-existing file.

Answer 2

A quick search on pypi brings up this which might do what you want, but otherwise I'd suggest writing it yourself, it would be a fairly simple tools to write.

I haven't tested it, but here's a very simple implementation that should do it (python3).

class RotatingFile:

    def __init__(self, basename, size, binary=False):
        self.basename = basename
        self.size = size
        self.counter = 0
        if binary:
            self.buffer = b''
        else:
            self.buffer = ''

    def write(self, data)
        self.buffer += data
        if len(self.buffer) >= self.size:
            data = self.buffer[:self.size]
            self.buffer = self.buffer[self.size:]
            name = self.basename + str(self.counter)
            with open(name) as f:
                f.write(data)
            self.counter += 1

    def flush(self):
        name = self.basename + str(self.counter)
        with open(name) as f:
            f.write(self.buffer)

So this should write to 6 files:

>>> f = RotatingFile('myfile', 1000)
>>> f.write('a' * 5500)
>>> f.flush()