A way in Python to agnostically append() / add() to a collection (or other receiver)?

Question

Is there a way in Python to add agnostically to a collection?

Given the prevalence of duck typing I was surprised that the method to add to a list is append(x) but the method to add to a set is add(x) .

I'm writing a family of utility functions that need to build up collections and would ideally like them not to care what type is accumulating the result. It should at least work for list and set - and ideally for other targets, as long as they know what method to implement. Essentially, the duck type here is 'thing to which items can be added'.

In practice, these utility functions will either be passed the target object to add the results to, or - more commonly - a function that generates new instances of the target type when needed.

For example:

def collate(xs, n, f_make=lambda: list()):
    if n < 1:
        raise ValueError('n < 1')
    col = f_make()
    for x in xs:
        if len(col) == n:
            yield col
            col = f_make()
        col.append(x)  # append() okay for list but not for set
    yield col

>>> list(collate(range(6), 3))
[[0, 1, 2], [3, 4, 5]]

>>> list(collate(range(6), 4))
[[0, 1, 2, 3], [4, 5]]

>>> # desired result here: [{1, 2, 3, 4}, {5, 6}]
>>> list(collate(range(6), 4, f_make=lambda: set()))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/paul/proj/mbrain/src/fossil/fn.py", line 42, in collate
    col.append(x)
AttributeError: 'set' object has no attribute 'append'

Here collate() is just a simple example. I expect there's already a way to achieve this 'collation' in Python. That's not the real question here.

I'm currently using Python 3.8.5.

Answer 1

(This is an edited answer. Feel free to look at history for my old answer, but it was not relevant to the question)

The pythonic way is to use the standard library here. Rather than manipulating lists, you can use some of the built-in functions that work on iterables more generally. As in, itertools .

This function is a rough guideline for using itertools . It doesn't handle the case where f_make() isn't blank. It's also a bit dense and if you're not used to python you probably won't find this super easy to read, but it's technically probably one of the more pythonic ways to do this. I'm not certain I'd recommend using this, but it does a lot in a fairly small number of lines, which is sort of the point. I'm sure someone else could find a "more pythonic" approach though.

def collate(xs, n, f_make=list):
    result = f_make()
    for _, val in itertools.groupby(
        enumerate(xs),
        lambda i: (len(result) + i[0]) // n
    ):
        yield list(itertools.chain(result, (v[1] for v in val)))

Edit 2

Your question has been edited so I'll address the clear points in there now:

If you want a duck-typed way to add to an iterable, you should probably create your own data structure. That way, you can handle all iterables, and not just sets and lists. If you pass a generator in, or something that's been sorted or a map result, you probably want to be able to handle those too, right?

Here's an example of a wrapper for that kind of thing:

class Appender:
  def __init__(self, iterable):
    self.length = sum(1 for _ in iterable)
    self.iterable = iter(iterable)

  def append(self, new_item):
    self.length += 1
    self.iterable = itertools.chain(self.iterable, new_item)

  def __iter__(self):
    return self.iterable

  def __len__(self):
    return self.length

Note that you can further modify this to be a MutableSequence but I don't think that's strictly necessary for your use case, where you just need length. If you don't care about iterables, then I'd advise you change your question title to remove "or other receivers"

Also note that this doesn't handle set s like set s (obviously). I'm of the belief that it should be up to the caller to manage the output of a function. I personally feel that it's perfectly acceptable to require that a function caller only pass in a MutableSequence , and responsibility of casting it to a set should be separate. This leads to clearer and more concise functions that require less logic. If you expect a set and/or dict is going to be a common acceptance method, it's likely worth handling that separately. As was mentioned in comments to your question, these are fundamentally different data types (particularly sets which are not ordered and thus can't really be collated without first being sorted into a non-set anyway).

Answer 2

Returning to this later I found a better solution using @functools.singledispatch which is also user-extensible to additional types.

import functools

@functools.singledispatch
def append(xs, v):
    raise ValueError('append() not supported for ' + str(type(xs)))


@append.register
def _(xs: MutableSequence, v):
    xs.append(v)


@append.register
def _(xs: MutableSet, v):
    xs.add(v)

Answer 3

Here's the solution I ended up with...

def appender(xs):
    if isinstance(xs, MutableSequence):
        f = xs.append
    elif isinstance(xs, MutableSet):
        f = xs.add
    # Could probably do better validation here...
    elif hasattr(xs, 'append'):  
        f = getattr(xs, 'append')
    else:
        raise ValueError('Don\'t know how to append to ' + str(type(xs)))
    return f


def collate(xs, n, f_make=lambda: list()):
    if n < 1:
        raise ValueError('n < 1')
    col = f_make()
    app = appender(col)
    for x in xs:
        if len(col) == n:
            yield col
            col = f_make()
            app = appender(col)
        app(x)
    if col:
        yield col

>>> list(collate(range(6), 4, set))
[{0, 1, 2, 3}, {4, 5}]

>>> list(collate(range(6), 4, list))
[[0, 1, 2, 3], [4, 5]]

(I previously added this to the question - and it was removed. So I'm now adding it as an answer.)

Additionally, just to clarify the intended behaviour:

>>> list(collate(range(6), 2, list))
[[0, 1], [2, 3], [4, 5]]

>>> list(collate(range(6), 1, set))
[{0}, {1}, {2}, {3}, {4}, {5}]