使用Postgres优化Django queryset查询

Question

I'm using Django 2.1 and python 3.6 I have a query that contains other queries and seems to take too long to evaluate when there is a lot of data to process. Basically it reaches the db user timeout (10 seconds) so the query fails, and when I try it without timeout, I get the following error message from DB (postgres)after almost a minute:

ERROR:  canceling statement due to conflict with recovery
DETAIL:  User query might have needed to see row versions that must be removed.

My question is - how can I make the query more efficient? (all used fields are indexed in DB)

This is the query:

migration_set__qs = Migration.objects.filter(
        migration_id=migration_id,
        migration_version=migration_version,
        migration_data__generated_id__isnull=False
    ).values_list(
        'object_id',
        flat=True
    )

containers__qs = Container.objects.all().exclude(
            Q(id__in=migration_set__qs) | Q(created_at__gte=turned_on_date)
        )

limited_containers = containers__qs[0:10]
num_containers_processed += limited_containers.count()

The 'count()' triggers the evaluation of the query and there it breaks.

Answer 1

First thing I would attempt, would be to change the migration_set_qs into Subquery :

migration_set__qs = Migration.objects.filter(
    migration_id=migration_id,
    migration_version=migration_version,
    migration_data__generated_id__isnull=False
).values(
    'object_id',
)

containers__qs = Container.objects.all().exclude(
    Q(id__in=Subquery(migration_set__qs)) | Q(created_at__gte=turned_on_date)
)

limited_containers = containers__qs[0:10]
num_containers_processed += limited_containers.count()

Although I am not sure if it will solve all your problems. Please let me know it if made any difference, and if I will have any other ideas I will update this answer as well.