SQLAlchemy union_all and all() returning incorrect number of items

Question

For some reason when I'm using SQLAlchemy's union_all and .all() , it's returning the incorrect number of items.

As you can see below, I broke each one down to see where the error was. Does anyone have any idea why this would be happening?

>>> pn = PostNotification.query.filter_by(notified_id=1)
>>> cn = CommentNotification.query.filter_by(notified_id=1)
>>> pn.count()
4
>>> cn.count()
2
>>> u = pn.union_all(cn)
>>> u.count()
6
>>> all = u.all()
>>> len(all)
5

Here are my two models:

class NotificationMixin:
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String(150), nullable=False)
    read = db.Column(db.Boolean, default=False)
    created = db.Column(db.DateTime, index=True, default=datetime.utcnow)

    @declared_attr
    def notifier_id(cls):
        return db.Column(db.Integer, db.ForeignKey('user.id'))

    @declared_attr
    def notified_id(cls):
        return db.Column(db.Integer, db.ForeignKey('user.id'))


class PostNotification(db.Model, NotificationMixin):
    post_id = db.Column(db.Integer, db.ForeignKey('post.id'))
    comment_id = db.Column(db.Integer)

    def __repr__(self):
        return '<PostNotification {}>'.format(self.name)


class CommentNotification(db.Model, NotificationMixin):
    post_id = db.Column(db.Integer, db.ForeignKey('post.id'))
    comment_id = db.Column(db.Integer, db.ForeignKey('post_comment.id'))

    def __repr__(self):
        return '<CommentNotification {}>'.format(self.name)

UPDATE :

Here is a screenshot of the data that represents the two models

When I define the columns explicitly, there is no issue when I'm using union_all . It only returns the incorrect amount of records when I db.session.query(PostNotification) and db.session.query(CommentNotification) .

pn = db.session.query(
    PostNotification.id,
    PostNotification.name,
    PostNotification.read,
    PostNotification.created,
    PostNotification.post_id,
    PostNotification.comment_id,
    PostNotification.notifier_id,
    PostNotification.notified_id).filter_by(
        notified_id=1)

cn = db.session.query(
    CommentNotification.id,
    CommentNotification.name,
    CommentNotification.read,
    CommentNotification.created,
    CommentNotification.post_id,
    CommentNotification.comment_id,
    CommentNotification.notifier_id,
    CommentNotification.notified_id).filter_by(
        notified_id=1)

u = pn.union_all(cn).order_by(PostNotification.created.desc())

>>> pn.count()
4
>>> cn.count()
2
u.count()
6
>>> all = u.all()
>>> len(all)
6

The problem with this, is I lose the model, and my relationships are gone. Therefore, I have to use this very ugly workaround. This only makes sense if you see the data in https://i.stack.imgur.com/UHfo7.jpg .

result = []
for row in u:
    if 'post' in row.name.split('_'):
        n = PostNotification.query.filter_by(id=row.id).first()
        result.append(n)
    if 'comment' in row.name.split('_'):
        n = CommentNotification.query.filter_by(id=row.id).first()
        result.append(n)

Now my result is in descending order, both tables are combined via union_all , and my relationships are back in tact. The problem now is, I obviously can't use result.paginate, because result is now a list .

Answer 1

The union u is not polymorphic in the sense that it'd recognize which rows represent PostNotification and which CommentNotification entities – it simply treats all rows as representing the primary entity PostNotification .

It also happens that you have 2 "identical" notifications in both tables, ie they have the same numeric value for primary key. SQLAlchemy deduplicates model entities based on primary key when querying, as noted here by the author of SQLAlchemy , and so len(u.all()) returns fewer results. u.count() on the other hand counts in the database and so counts all rows. This deduplication does not take place if querying attributes or more than 1 entity.

Answer 2

Looks like I figured it out. I can now query AbstractNotification directly db.session.query(AbstractNotification).all()

from sqlalchemy.ext.declarative import AbstractConcreteBase   


class AbstractNotification(AbstractConcreteBase, db.Model):
    __table__ = None


class NotificationBaseModel:
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String(150), nullable=False)
    read = db.Column(db.Boolean, default=False)
    created = db.Column(db.DateTime, index=True, default=datetime.utcnow)

    @declared_attr
    def notifier_id(cls):
        return db.Column(db.Integer, db.ForeignKey('user.id'))

    @declared_attr
    def notified_id(cls):
        return db.Column(db.Integer, db.ForeignKey('user.id'))


class PostNotification(AbstractNotification, NotificationBaseModel):
    post_id = db.Column(db.Integer, db.ForeignKey('post.id'))
    comment_id = db.Column(db.Integer)

    __mapper_args__ = {
        'polymorphic_identity': 'post_notification',
        'concrete': True
        }

    def __repr__(self):
        return '<PostNotification {}>'.format(self.name)


class CommentNotification(AbstractNotification, NotificationBaseModel):
    post_id = db.Column(db.Integer, db.ForeignKey('post.id'))
    comment_id = db.Column(db.Integer, db.ForeignKey('post_comment.id'))

    __mapper_args__ = {
        'polymorphic_identity': 'comment_notification',
        'concrete': True
        }

    def __repr__(self):
        return '<CommentNotification {}>'.format(self.name)