如何使用Django ORM进行批量插入或增量类型操作

Question

I have a model as defined here:

class VectorSet(models.Model):
    word = models.CharField(max_length=255)
    weight = models.IntegerField()
    session = models.ForeignKey(ResearchSession)

I want to write a function that will take a list of words and a ResearchSession, and for each word in that list of words if it's doesn't already exist, create a new row with a weight of 1, otherwise take that row and increment weight by 1.

So far I've gotten this:

def train(words, session):
    for i in words:
        result, created = VectorSet.objects.get_or_create(word=i, session=session,
                                                          defaults={'weight' : 1})
        if not created:
            result.weight = F('weight') + 1
            result.save()

I'm fairly confident that there is a way to do this with one query however I can't quite figure out what that might be or if it's possible to do with django code over raw SQL.

Answer 1

There is currently no out-of-the box solution for doing bulk inserts other than bulk_create I think. Another solution, depending on your database, is to perform get_or_create within a transaction by using atomic . For example:

from django.db import transaction

@transaction.atomic
def train(words, session):
    for i in words:
        result, created = VectorSet.objects.get_or_create(word=i, session=session,
                                                      defaults={'weight' : 1})
        if not created:
            result.weight = F('weight') + 1
            result.save()

Otherwise, you might be able to use the DB API executemany :

cursor.executemany('INSERT INTO vectorset (field1, field2, field3) VALUES (?, ?, ?)', data)

Answer 2

Logic is simple, but we need to hit DB several times, which means several queries:

qs = VectorSet.objects.filter(word__in=words, session=session)
qs.update(weiget=models.F('weight')+1)
VectorSet.objects.bulk_insert(VectorSet(session=session, word=w, weight=1)
  for w in words if w not in qs.value_list('word', flat=True))

There is also a update_or_create in Django 1.7, but currently it does not distinguish defaults for update from defaults for create:

for w in words:
    VectorSet.objects.update_or_create(word=w, session=session,
                                    defaults={'weight': models.F('weight')+1})

Thus the above code will fail in creating by int(models.F('weight')+1) (We could override the __int__ method, but too hack to make sense...IMO)