Django custom filter in admin using multiple databases

Question

I have multiple databases, one Django managed and one external containing relevant information for filtering down inside Django admin using SimpleListFilter. As I don't have foreign keys across databases due to limitations in Django, I'm doing a lookup in the external database to fetch for example a target version number. Based on that lookup list I am able to reduce my queryset.

Now the problem is that my database is too large to filter down that way, as the resulting SQL query looks like the following:

SELECT 'status'.'id', 'status'.'service_number', 'status'.'status'
    FROM 'status'
    WHERE ('status'.'service_number' = '01xxx'  OR 'status'.'service_number' = '02xxx'  OR 'status'.'service_number' = '03xxx' ......

The list of OR's is too long and the reduce cannot be done in the database anymore, the error received is:

Django Version: 1.4.4
Exception Type: DatabaseError
Exception Value:    (1153, "Got a packet bigger than 'max_allowed_packet' bytes")

I increased already max_allowed_packet in MySQL, but this time I don't think it is the right way to simply increase that value again.changed

My SimpleListFilter looks like:

class TargetFilter(SimpleListFilter):
    parameter_name = 'target'

    def lookups(self, request, model_admin):
        return (
            ('v1', 'V1.0'),
            ('v2', 'V2.0'),
        )

    def queryset(self, request, queryset):
        if self.value():
            lookup = []
            for i in Target.objects.using('externaldb').filter(target=self.value()).values('service_number').distinct():
                lookup.append(str(i['service_number']))
            qlist = [Q(**{'service_number': f}) for f in lookup]
            queryset = queryset.filter(reduce(operator.or_, qlist))
            return queryset

The listed code worked for years, but became fast slower and now isn't working at all. I've tried to use frozensets, but this doesn't seem to work. Do you have an idea on how I can reduce very large sets?

Thanks for any hint!

Answer 1

A little too late to the game, but I just started working with Django a couple of weeks ago and I had huge tables to work with. For large queries I used RAW Sql in Django and cursors as a matter of fact to limit the returned output, and process it in batches. Querysets get everything and loads it in memory which is not practical in large deployments.

Look at https://docs.djangoproject.com/en/3.0/topics/db/sql/ and more specifically Executing custom SQL directly