如何区分对象等文件与对象等文件路径

Question

Summary:

There is a variety of function for which it would be very useful to be able to pass in two kinds of objects: an object that represents a path (usually a string), and an object that represents a stream of some sort (often something derived from IOBase , but not always). How can this variety of function differentiate between these two kinds of objects so they can be handled appropriately?

---

Say I have a function intended to write a file from some kind of object file generator method:

spiff = MySpiffy()

def spiffy_file_makerA(spiffy_obj, file):
    file_str = '\n'.join(spiffy_obj.gen_file()) 
    file.write(file_str)

with open('spiff.out', 'x') as f:
    spiffy_file_makerA(spiff, f)
    ...do other stuff with f...

This works. Yay. But I'd prefer to not have to worry about opening the file first or passing streams around, at least sometimes... so I refactor with the ability to take a file path like object instead of a file like object, and a return statement:

def spiffy_file_makerB(spiffy_obj, file, mode):
    file_str = '\n'.join(spiffy_obj.gen_file()) 
    file = open(file, mode)
    file.write(file_str)
    return file

with spiffy_file_makerB(spiff, 'file.out', 'x') as f:
    ...do other stuff with f...

But now I get the idea that it would be useful to have a third function that combines the other two versions depending on whether file is file like, or file path like, but returns the f destination file like object to a context manager. So that I can write code like this:

with  spiffy_file_makerAB(spiffy_obj, file_path_like, mode = 'x') as f:
    ...do other stuff with f...

...but also like this:

file_like_obj = get_some_socket_or_stream()

with spiffy_file_makerAB(spiffy_obj, file_like_obj, mode = 'x'):
    ...do other stuff with file_like_obj...
    # file_like_obj stream closes when context manager exits 
    # unless `closefd=False` 

Note that this will require something a bit different than the simplified versions provided above.

Try as a I might, I haven't been able to find an obvious way to do this, and the ways I have found seem pretty contrived and just a potential for problems later. For example:

def spiffy_file_makerAB(spiffy_obj, file, mode, *, closefd=True):
    try: 
        # file-like (use the file descriptor to open)
        result_f = open(file.fileno(), mode, closefd=closefd)
    except TypeError: 
        # file-path-like
        result_f = open(file, mode)
    finally: 
        file_str = '\n'.join(spiffy_obj.gen_file()) 
        result_f.write(file_str)
        return result_f

Are there any suggestions for a better way? Am I way off base and need to be handling this completely differently?

Answer 1

For my money, and this is an opinionated answer, checking for the attributes of the file-like object for the operations you will need is a pythonic way to determine an object's type because that is the nature of pythonic duck tests/duck-typing :

Duck typing is heavily used in Python, with the canonical example being file-like classes (for example, cStringIO allows a Python string to be treated as a file).

Or from the python docs' definition of duck-typing

A programming style which does not look at an object's type to determine if it has the right interface; instead, the method or attribute is simply called or used (“ If it looks like a duck and quacks like a duck, it must be a duck. ”) By emphasizing interfaces rather than specific types, well-designed code improves its flexibility by allowing polymorphic substitution. Duck-typing avoids tests using type() or isinstance() . (Note, however, that duck-typing can be complemented with abstract base classes.) Instead, it typically employs hasattr() tests or EAFP programming.

If you feel very strongly that there is some very good reason that just checking the interface for suitability isn't enough, you can just reverse the test and test for basestring or str to test whether the provided object is path-like. The test will be different depending on your version of python.

is_file_like = not isinstance(fp, basestring) # python 2
is_file_like = not isinstance(fp, str) # python 3

In any case, for your context manager, I would go ahead and make a full-blown object like the below in order to wrap the functionality that you were looking for.

class SpiffyContextGuard(object):
    def __init__(self, spiffy_obj, file, mode, closefd=True):
        self.spiffy_obj = spiffy_obj
        is_file_like = all(hasattr(attr) for attr in ('seek', 'close', 'read', 'write'))
        self.fp = file if is_file_like else open(file, mode)
        self.closefd = closefd

    def __enter__(self):
        return self.fp

    def __exit__(self, type_, value, traceback):
        generated = '\n'.join(self.spiffy_obj.gen_file())
        self.fp.write(generated)
        if self.closefd:
            self.fp.__exit__()

And then use it like this:

with SpiffyContextGuard(obj, 'hamlet.txt', 'w', True) as f:
    f.write('Oh that this too too sullied flesh\n')

fp = open('hamlet.txt', 'a')
with SpiffyContextGuard(obj, fp, 'a', False) as f:
    f.write('Would melt, thaw, resolve itself into a dew\n')

with SpiffyContextGuard(obj, fp, 'a', True) as f:
    f.write('Or that the everlasting had not fixed his canon\n')

If you wanted to use try/catch semantics to check for type suitability, you could also wrap the file operations you wanted to expose on your context guard:

class SpiffyContextGuard(object):
    def __init__(self, spiffy_obj, file, mode, closefd=True):
        self.spiffy_obj = spiffy_obj
        self.fp = self.file_or_path = file 
        self.mode = mode
        self.closefd = closefd

    def seek(self, offset, *args):
        try:
            self.fp.seek(offset, *args)
        except AttributeError:
            self.fp = open(self.file_or_path, mode)
            self.fp.seek(offset, *args)

    # define wrappers for write, read, etc., as well

    def __enter__(self):
        return self

    def __exit__(self, type_, value, traceback):
        generated = '\n'.join(self.spiffy_obj.gen_file())
        self.write(generated)
        if self.closefd:
            self.fp.__exit__()

Answer 2

Probably not the answer you're looking for, but from a taste point of view I think it's better to have functions that only do one thing. Reasoning about them is easier this way.

I'd just have two functions: spiffy_file_makerA(spiffy_obj, file) , which handles your first case, and a convenience function that wraps spiffy_file_makerA and creates a file for you.

Answer 3

my suggestion is to pass pathlib.Path objects around. you can simply .write_bytes(...) or .write_text(...) to these objects.

other that that you'd have to check the type of your file variable (this is how polymorphism can be done in python):

from io import IOBase

def some_function(file)
    if isinstance(file, IOBase):
        file.write(...)
    else:
        with open(file, 'w') as file_handler:
            file_handler.write(...)

(i hope io.IOBase is the most basic class to check against...). and you would have to catch possible exceptions around all that.

Answer 4

Another approach to this problem, inspired by this talk from Raymond Hettinger at PyCon 2013 , would be to keep the two functions separate as suggested by a couple of the other answers, but to bring the functions together into a class with a number of alternative options for outputting the object.

Continuing with the example I started with, it might look something like this:

class SpiffyFile(object):
    def __init__(self, spiffy_obj, file_path = None, *, mode = 'w'):
        self.spiffy = spiffy_obj
        self.file_path = file_path
        self.mode = mode
    def to_str(self):
        return '\n'.join(self.spiffy.gen_file())
    def to_stream(self, fstream):
        fstream.write(self.to_str())
    def __enter__(self):
        try:
            # do not override an existing stream
            self.fstream
        except AttributeError:
            # convert self.file_path to str to allow for pathlib.Path objects
            self.fstream = open(str(self.file_path), mode = self.mode)
        return self
    def __exit__(self, exc_t, exc_v, tb):
        self.fstream.close()
        del self.fstream
    def to_file(self, file_path = None, mode = None):
        if mode is None:
            mode = self.mode
        try:
            fstream = self.fstream
        except AttributeError:
            if file_path is None:
                file_path = self.file_path
            # convert file_path to str to allow for pathlib.Path objects
            with open(str(file_path), mode = mode) as fstream:
                self.to_stream(fstream)
        else:
            if mode != fstream.mode:
                raise IOError('Ambiguous stream output mode: \
                           provided mode and fstream.mode conflict')
            if file_path is not None:
                raise IOError('Ambiguous output destination: \
                           a file_path was provided with an already active file stream.')
            self.to_stream(fstream)

Now we have lots of different options for exporting a MySpiffy object by using a SpiffyFile object. We can just write it to a file directly:

from pathlib import Path
spiff = MySpiffy()
p = Path('spiffies')/'new_spiff.txt'
SpiffyFile(spiff, p).to_file()

We can override the path, too:

SpiffyFile(spiff).to_file(p.parent/'other_spiff.text')

But we can also use an existing open stream:

SpiffyFile(spiff).to_stream(my_stream)

Or, if we want to edit the string first we could open a new file stream ourselves and write the edited string to it:

my_heading = 'This is a spiffy object\n\n'
with open(str(p), mode = 'w') as fout:
    spiff_out = SpiffyFile(spiff).to_str()
    fout.write(my_heading + spiff_out)

And finally, we can just use a context manager with the SpiffyFile object directly to as many different locations- or streams- as we like (note that we can pass the pathlib.Path object directly without worrying about string conversion, which is nifty):

with SpiffyFile(spiff, p) as spiff_file:
    spiff_file.to_file()
    spiff_file.to_file(p.parent/'new_spiff.txt')
    print(spiff_file.to_str())
    spiff_file.to_stream(my_open_stream)

This approach is more consistent with the mantra: explicit is better than implicit.