Filtering a Django queryset's related ManyToMany field when using values()

Question

I'm working with a queryset which includes a ManyToMany field brand_groups . Users only have access to a subset of BrandGroups based on their organization. I'm trying to keep that ManyToMany field filtered down while still using values() , which is heavily integrated with the view.

The simplified tables I'm working with:

class BrandGroup(models.Model):
    id = models.BigAutoField(primary_key=True)
    name = models.CharField(max_length=256)
    organization = models.ForeignKey(
        Organization, related_name="brand_groups", null=False
    )

class Fact(models.Model):
    id = models.CharField(max_length=256, primary_key=True)
    brand_groups = models.ManyToManyField(BrandGroup, blank=True)

What's worked for me in the past is using Prefetch objects to handle this kind of limiting:

qs = Fact.objects.prefetch_related(
    Prefetch("brand_groups",
        queryset=BrandGroup.objects.filter(
            organization_id=self.request.META["ORG_ID_HEADER"]
)))

But I find that values() seems to ignore prefetches entirely.

qs.values("brand_groups__name")

The above always includes the full set of associated BrandGroup objects without the filter.

I've tried adding to_attr='org_brand_groups' to the Prefetch , but then qs.values("org_brand_groups__name") complains that the field doesn't exist.

I've also tried using an annotation to rename the prefetched field in a similar way. I don't get a complaint about the field not existing, but again values() returns the unfiltered queryset.

The only way I've managed to accomplish this kind of filtering is by using a subquery:

qs = Fact.objects.annotate(
    brand_group_name=Subquery(
        BrandGroup.objects.filter(
            organization_id=self.request.META["ORG_ID_HEADER"],
            Q(id=OuterRef("brand_groups__id"))).values(
                "name"[:1],output_field=CharField(),))

# Now it gives me the desired results
qs.values("brand_group_name")

But this approach negates what I'm trying to accomplish. The goal is to pull the BrandGroup data in using a join, not a subquery.

Is there any method of filtering a ManyToMany related field without subqueries that will work with values() ? My only remaining idea is to filter the queryset with Python after values() is already applied.

Answer 1

I think you misunderstand what prefetch_related does:

prefetch_related, on the other hand, does a separate lookup for each relationship, and does the 'joining' in Python. This allows it to prefetch many-to-many and many-to-one objects, which cannot be done using select_related, in addition to the foreign key and one-to-one relationships that are supported by select_related.

Your best bet is to use a subquery like in your example.

Answer 2

Ultimately, I ended up not doing any filtering before using values and using an Exists filter after values was called.

qs = Fact.objects.filter(...)
qs = qs.values("brand_groups__name", "brand_groups__id")

qs = qs.annotate(
    brand_group_accessible=Exists(
        BrandGroup.objects.filter(
            organization_id=self.request.META["ORG_ID_HEADER"],
            Q(id=OuterRef("brand_groups__id")))))

qs = qs.filter(Q(brand_group_accessible=True) | Q(brand_groups__id__isnull=True))