Merge multiple files into a single stream in Python

Question

I have a dozen of files that I would like to present to the user as a single read only file like object. I do not want to load them into memory at once, nor merge them in the filesystem. I would like something like a itertools.chain of mmap.mmap but present the API of a file like object (ie with the file methods like read, etc). Is this possible?

Answer 1

You can use fileinput module here to read through multiple files.

Lets say for example you want to read two files new.txt and IQ.txt .

for line in fileinput.input(["C:\\Users\\Administrator\\Desktop\\new.txt","C:\\Users\\Administrator\\Desktop\\IQ.txt"]):
print line,

In nutshell you provide a list of files you want to read and do it.

Answer 2

I have had some success with the SplitFileReader function from here :

from split_file_reader import SplitFileReader

filepaths = [...]

with SplitFileReader(filepaths, mode="rb") as fin:
    fin.read(...)

The builtin fileinput can only be used for text files, because it does not implement the read function.

The splitfile function from here probably works, too, but it has its own convention of how the partial files must be named, and does not simply accept a list of paths.

I am surprised I could not find a builtin function for such a fundamental task, or a package developed by a bigger community.