Django-非常慢的查询

Question

I've made a module which parses xml file and updates or creates data in django db (pgsql).

When the data import/update is done I try to update some meta data of my objects.

I use django-mptt for tree structures and my meta-data updater is for creating such structures between my objects.

It's really really slow it takes about 1 second to populate parent with data from other foreignkey.

How do I optimise this?

for index, place in enumerate(Place.objects.filter(type=Place.TOWN, town_id_equal=True)):
    place.parent = place.second_order_division
    place.save()

    print index
    if index % 5000 == 0:
        transaction.commit()
transaction.commit()

transaction.set_autocommit(False)
for index, place in enumerate(Place.objects.filter(type=Place.TOWN, town_id_equal=False,
                                                   parent__isnull=True)):

    place.parent = Place.objects.get(town_id=place.town_id_extra)
    place.save()

    print index
    if index % 5000 == 0:
        transaction.commit()
transaction.commit()


class Place(MPTTModel):
    first_order_division = models.ForeignKey("self", null=True, blank=True, verbose_name=u"Województwo",
                                             related_name="voivodeships")
    second_order_division = models.ForeignKey("self", null=True, blank=True, verbose_name=u"Powiat",
                                              related_name="counties")
    parent = TreeForeignKey('self', null=True, blank=True, related_name='children')

Edit:

I updated first function like this:

transaction.set_autocommit(False)
for index, obj in enumerate(Place.objects.filter(type=Place.COUNTY)):
    data = Place.objects.filter(second_order_division=obj, type=Place.TOWN, town_id_equal=True)
    data.update(parent=obj)
    print index
    transaction.commit()

Answer 1

Instead of using loop you should do bulk updates like

for first transaction you can replace your transaction with this Django query:

Place.objects.filter(type=Place.TOWN, town_id_equal=True).update(parent=F('second_order_division'))

For second transaction we can not apply bulk update because of again query on Place model. for this you should do something to save hitting 'Place.objects.get(town_id=place.town_id_extra)' query each time in loop.

or can take help from this blog

Answer 2

Answering a more general question, one tactic to improve performance of almost any type of system is:

Minimize interaction between the dynamic parts of your system

That's it: minimize interaction through HTTP requests, database queries, etc. In your case, you are doing multiple queries to your database that can be easily reduced to fewer (perhaps one or two).