Django ORM 注释查找有多少对象有任何相关 object

Question

I am currently trying to find out what % of a set of objects have a related object with certain values. Specifically, I have a table of objects and a one to many relationship to a table of comments, and I am trying to figure out what percentage of those objects have comments in a specific length. Both of these tables are ETL output from a separate dataset to allow easier metric calculations.

# Models
class Data(models.Model):
    id = models.AutoField(primary_key=True)
    creator_id = models.IntegerField() # Not a real foreign key

class DataCommenter(models.Model):
    do_id = models.ForeignKey(Data)
    creator_id = models.IntegerField() # Not a real foreign key
    short_comments = models.IntegerField()
    medium_comments = models.IntegerField()
    long_comments = models.IntegerField()

From these models, I have some queryset annotations that are being performed to try and get the average as shown below:

# QuerySet
class DataQuerySet(models.QuerySet):
    def extensive_comments(self):
        """Get extensive comment raw selection."""
        inner_query = DataCommenter.objects.exclude(
            creator_id=OuterRef("creator_id")
        ).filter(
            Q(medium_comments__gte=1) | Q(long_comments__gte=1), do_id=OuterRef("id")
        )
        return self.annotate(
            raw_extensive_comments=Case(
                When(
                    Exists(inner_query), then=1
                ), default=0, output_field=FloatField()
            )
        )

    def annotate_period(self):
        """Annotation to allow average without aggregation."""
        return self.annotate(period=Value(value=True, output_field=BooleanField()))

The QuerySet is attached to the Data model and is used as follows:

Data.objects.all().annotate_period().extensive_comments().values("period").annotate(
    extensive_comments=ExpressionWrapper(Avg(raw_extensive_comments) * 100, output_field=FloatField())
)

The data that we have includes multiple DataCommenter objects for some Data objects, and for some reason the average runs against the number of DataCommenter objects instead of the number of Data objects, so what should be 3/5 data objects yielding a 60, we get something like 10/12 and get 83.33333. We have a large number of other metrics that we are calculating using a number of other fields not shown here, so we can't use aggregate , and the use of the values("period") should have the objects be treated as a single group for the later annotations that include the Avg , and they do for every metric but this single calculation. We have tried having the inner_query be inside of the annotation directly, tried having that inner query have .values("do_id").distinct() at the end, tried to remove the query entirely and operate using commenter__medium_comments type filtering directly in the Data model, and I have no idea why it is returning like this. Any help would be greatly appreciated.

Answer 1

The annotations listed in the question are completely correct and the actual problem was with the FilterSet in the view itself. Specifically, we had a set of filters looking at creator_id like the following (related name for the DataCommenter model to Data is commenter ):

class BaseAggregateFilterSet(filters.FilterSet):
    creator_id = filters.NumberFilter(method=filter_by_creator_id")

    def filter_by_creator_id(self, qs, _, value):
        return qs.filter(Q(creator_id=value) | Q(commenter__creator_id=value)).distinct()

I thought that the distinct() at the end of that query would ensure that if you have 1 Data object with a DataCommenter object with the same creator_id, or even one with a different one that matched the selection, you would get back only the singular rows for the Data objects. What actually happens is that there is an implicit INNER JOIN done that multiplies each of your Data objects by the number DataCommenter objects. The correct way to do this to ensure that you maintain your QuerySet of only Data objects is to structure it like the following:

def filter_by_creator_id(self, qs, _, value):
    data_ids = qs.filter(Q(creator_id=value) | Q(commenter__creator_id=value)).values("id").distinct()
    return qs.filter(id__in=data_ids)

Once we made that change, all of our annotations that were returning strange results for averages were fixed as they were now operating on the expected number of objects instead of some objects counting multiple times.