Django prefetch_related GenericForeignKey with multiple content types

Question

I'm using django-activity-stream to display a list of recent events. For the sake of example these could be someone commenting or someone editing an article. Ie the GenericForeignKey action_object could reference a Comment or an Article . I'd like to display a link to whatever the action_object is too:

<a href="{{ action.action_object.get_absolute_url }}">
{{ action.action_object }}
</a>

The problem is this causes queries for every single item, particularly as Comment.get_absolute_url requires the comment's article , which has not been fetched yet, and Article.__unicode__ requires its revision.content , which also hasn't been fetched.

django-activity-stream already calls prefetch_related('action_object') automatically ( related discussion ). This appears to be working as testing with {{ action.action_object.id }} results in a single query per action_object_content_type , despite the docs saying:

It also supports prefetching of GenericRelation and GenericForeignKey, however, it must be restricted to a homogeneous set of results. For example, prefetching objects referenced by a GenericForeignKey is only supported if the query is restricted to one ContentType.

And there is more than one content type. However in my use case above I need extra prefetch_related calls, for example:

query = query.prefetch_related('action_object__article`, `action_object__revision`)

But this complains because Article s don't have an __article (and would probably complain about Comment s not having a __revision too if it got that far). I'm assuming this is what the docs are really referring to. So I thought I'd try this:

comments = query._clone().filter(action_object_content_type=comment_ctype).prefetch_related('action_object__article')
articles = query._clone().filter(action_object_content_type=article_ctype).prefetch_related('action_object__revision')
query = comments | articles

But the results are always empty. I guess querysets only support a single prefetch_related list and can't be joined like that.

I like a single queryset to return because further filtering is done later in the code which this part doesn't know about. Although once the queryset is finally evaluated I want to be able to have django fetch everything needed to render the events.

Is there another way?

I had a look at Prefetch objects but I don't think they offer any help in this situation.

Answer 1

A solution can be found in django-notify-x which is derived from django-notifications which, in turn, is derived from django-activity-stream . It makes use of a "django snippet" linked in the copied text below.

https://github.com/v1k45/django-notify-x/pull/19

Using a snippet from https://djangosnippets.org/snippets/2492/ , prefetch generic relations to reduce the number of queries.

Currently, we trigger one additional query for each generic relation for each record, with this code, we reduce to one additional query for each generic relation for each type of generic relation used.

If all your notifications are related to a Badges model, only one aditional query will be triggered.

Answer 2

For Django 1.10 and 1.11, I am using the snippet above modified as below (just in case you are not using django-activity-stream):

from django.contrib.contenttypes.models import ContentType
from django.contrib.contenttypes import fields as generic


def get_field_by_name(meta, fname):
    return [f for f in meta.get_fields() if f.name == fname]


def prefetch_relations(weak_queryset):
    weak_queryset = weak_queryset.select_related()

    # reverse model's generic foreign keys into a dict:
    # { 'field_name': generic.GenericForeignKey instance, ... }
    gfks = {}
    for name, gfk in weak_queryset.model.__dict__.items():
        if not isinstance(gfk, generic.GenericForeignKey):
            continue
        gfks[name] = gfk

    data = {}
    for weak_model in weak_queryset:
        for gfk_name, gfk_field in gfks.items():
            related_content_type_id = getattr(weak_model, get_field_by_name(gfk_field.model._meta, gfk_field.ct_field)[
                0].get_attname())
            if not related_content_type_id:
                continue
            related_content_type = ContentType.objects.get_for_id(related_content_type_id)
            related_object_id = int(getattr(weak_model, gfk_field.fk_field))

            if related_content_type not in data.keys():
                data[related_content_type] = []
            data[related_content_type].append(related_object_id)

    for content_type, object_ids in data.items():
        model_class = content_type.model_class()
        models = prefetch_relations(model_class.objects.filter(pk__in=object_ids))
        for model in models:
            for weak_model in weak_queryset:
                for gfk_name, gfk_field in gfks.items():
                    related_content_type_id = getattr(weak_model,
                                                      get_field_by_name(gfk_field.model._meta, gfk_field.ct_field)[
                                                          0].get_attname())
                    if not related_content_type_id:
                        continue
                    related_content_type = ContentType.objects.get_for_id(related_content_type_id)
                    related_object_id = int(getattr(weak_model, gfk_field.fk_field))

                    if related_object_id != model.pk:
                        continue
                    if related_content_type != content_type:
                        continue

                    setattr(weak_model, gfk_name, model)

    return weak_queryset

This is giving me the intended results.

EDIT:

To use it, you simply call prefetch_relations, with your QuerySet as the argument.

For example, instead of:

my_objects = MyModel.objects.all()

you can do this:

my_objects = prefetch_relations(MyModel.objects.all())