Django: Annotate sum on variable number of columns

Question

So I've read the annotate columns, and utilizing the F() functions, and I saw this post on how to sum multiple columns. However, I'm working with an EAV style DB, so I'm given a variable list of columns. Consider the example:

class TestModel(models.Model):
    column_a = models.FloatField()
    column_b = models.FloatField()
    column_c = models.FloatField()
    column_d = models.FloatField()

ATTEMPT 1:

columns = {'column_a', 'column_c', 'column_d'}
queryset = DummyModel.objects.annotate(total=Sum([F(column_name) for column_name in columns]))

However, a print(queryset.query) yields the error django.core.exceptions.FieldError: Cannot resolve expression type, unknown output_field

ATTEMPT 2:

queryset = DummyModel.objects.annotate(total=ExpressionWrapper(Sum([F(column_name) for column_name in columns]), output_field=FloatField())) 

This does not yield a compilation error, but the SQL query yields:

SELECT "test_model"."column_a", "test_model"."column_c", "test_model"."column_d", SUM([]) AS "total" FROM "test_model"

Which is empty. Does anyone have any idea of how to solve this? Any help is greatly appreciated!

Answer 1

To figure this out, it helps to picture the table first:

column_a | column b | column_c
---------+----------+----------
1        | 2        | 3
4        | 5        | 6
7        | 8        | 9

Sum is a "vertical" operation; that is, if we wanted the sum of column_a , we could do

>>> DummyModel.objects.aggregate(total=Sum('column_a'))
{'total': 12}

As you can see, this returns 1 + 4 + 7 == 12 -- so you can see why I call it a "vertical" sum. Notice we use aggregate rather than annotate : aggregate is for vertical operators.

If instead we want "horizontal" sum -- the total across a row -- we'd use F() and + . So to get column_a + column_b in each row, we'd use

>>> DummyModel.objects.annotate(total=F('column_a') + F('column_b')).values('total')
<QuerySet [{'total': 3}, {'total': 9}, {'total': 15}]>

Hopefully you can see why I call this a "horizontal" sum: we're getting a sum of a and b "horizontally" in each row. And now notice we use annotate , which is for horizontal operations.

If the names of the columns aren't known beforehand, you'd need to get tricky and use functools.reduce and operator.add to build up an expression:

>>> from functools import reduce
>>> from operator import add

>>> cols = ['column_b', 'column_c']
>>> expr = reduce(add, (F(col) for col in cols))
>>> DummyModel.objects.annotate(total=expr).values('total')
<QuerySet [{'total': 5}, {'total': 11}, {'total': 17}]>

If we want both a horizontal and a vertical sum -- ie the sum of column_a plus the sum of column_b -- we need to use Sum and F() :

>>> DummyModel.objects.aggregate(total=Sum(F('column_a') + F('column_b')))
{'total': 27}

Notice: aggregate rather than annotate, since we're ultimately going with a vertical operation; a Sum of rows. Yes, there's a horizontal operation first, but since we ultimately Sum , we need aggregate .

So, to wrap things up, if the fields are a variable, we need to combine aggregate , Sum , and the reduce trickery from above:

>>> cols = ['column_b', 'column_c']
>>> expr = reduce(add, (F(col) for col in cols))
>>> DummyModel.objects.aggregate(total=Sum(expr))
{'total': 33}

Hope this helps!