如何在不评估的情况下拆分 Django 查询集？

Question

I am dealing with a Queryset of over 5 million + items (For batch ML purposes) and I need to split the queryset (so I can perform multithreading operations) without evaluating the queryset as I only ever need to access each item in the queryset once and thus I don't want to cache the queryset items which evaluating causes.

Is it possible to select the items into one queryset and split this without evaluating? or am I going to have to approach it by querying for multiple querysets using Limits [:size] to achieve this behaviour?

NB: I am aware that an Iterable can be used to cycle through a queryset without evaluating it but my question is related to how I can I split a queryset (if possible) to then run an iterable on each of the splitted querysets.

Answer 1

Django provides a few classes that help you manage paginated data – that is, data that's split across several pages, with “Previous/Next” links:

from django.core.paginator import Paginator

object_list = MyModel.objects.all()
paginator = Paginator(object_list, 10) # Show 10 objects per page, you can choose any other value

for i in paginator.page_range(): # A 1-based range iterator of page numbers, e.g. yielding [1, 2, 3, 4].
    data = iter(paginator.get_page(i))
    # use data

Answer 2

If your django version is 1.11 or less than that like 1.10 , 1.9 or so on, then use paginator.page(page_no) but be careful that this may raise an InvalidPage Exception when invalid/no page has been found.

For versions <= 1.11 , use below code:

from django.core.paginator import Paginator

qs = MyModel.objects.all()
paginator = Paginator(qs, 20)

for page_no in paginator.page_range:
    current_page = paginator.page(page_no)
    current_qs = current_page.object_list

And if you're using django version >= 2.0, please use paginator.get_page(page_no) instead, but you can also use paginator.page(page_no) .

For versions >= 2.0, use below code:

from django.core.paginator import Paginator

qs = MyModel.objects.all()
paginator = Paginator(qs, 20)

for page_no in paginator.page_range:
    current_page = paginator.get_page(page_no)
    current_qs = current_page.object_list

The advantage of using paginator.get_page(page_no) according to django documentations is as follows:

Return a valid page, even if the page argument isn't a number or isn't in range.

While in the case of paginator.page(page_no) , you have to handle the exception manually if page_no is not a number or is out of range.

Answer 3

Passing query sets to Threads is not something I would recommend. I know the sort of thing you are trying to do and why, but its best to just pass some sort of param set to each thread and then have the Thread perform the partial query. Working this way, your threads are distinct from the calling code.

On a different note, if you are trying to use threads as a work around for the lags caused by high DB queries, you might find using transaction management a better route. This link link has some useful tips. I use this instead of Threads

Answer 4

Yes you can, as from thisgist

Per the updated answer:

def queryset_iterator(queryset, chunk_size=1000):
"""
Iterate over a Django Queryset ordered by the primary key
This method loads a maximum of chunk_size (default: 1000) rows in it's
memory at the same time while django normally would load all rows in it's
memory. Using the iterator() method only causes it to not preload all the
classes.
Note that the implementation of the iterator does not support ordered query sets.
"""
    try:
        last_pk = queryset.order_by('-pk')[:1].get().pk
    except ObjectDoesNotExist:
        return

    pk = 0
    queryset = queryset.order_by('pk')
    while pk < last_pk:
        for row in queryset.filter(pk__gt=pk)[:chunk_size]:
            pk = row.pk
            yield row
        gc.collect()