Django - Following A Foreign Key To Another Foreign Key

Question

---

I'm trying to speedup lookups on my models. I have 3 models.

class sett(models.Model):
    setid = models.IntegerField(primary_key=True)
    block = models.ForeignKey(mapt, related_name='sett_block')

class data_parsed(models.Model):
    setid = models.ForeignKey(sett, related_name='data_parsed_setid', primary_key=True, db_index = True)
    setid_hash = models.CharField(max_length=100, db_index = True)

class unique_variables(models.Model):
    setid = models.ForeignKey(sett, related_name='unique_variables_setid', db_index = True)
    setid_hash = models.CharField(max_length=100, db_index = True)

---

Most importantly, sett has a primary key called setid, which:

data_parsed, and unique_variables have a foreign key to setid

data_parsed: 3 million rows

unique_variables: 13 million rows

---

Then I have some piece of code that tries to iterate through all data_parsed objects.

def something_1():
    timeNow = datetime.datetime.now()
    for i in range(0,N) <---- N is predefined, and no less than 10
        for contents in data_parsed.objects.all().iterator():
            pass
    timeEnd = datetime.datetime.now()
    print str(timeEnd - timeNow)

data_parsed has a few million items, and it takes around 20 seconds to go through, which I think is pretty fast.

---

Now, since setid is the same fields on both data_parsed, and unique_variables. I can get items in unique_variables with the foriegn key of data_parsed. Hence:

def something_2():
    timeNow = datetime.datetime.now()
    for i in range (0, N) <---- N is predefined, and no less than 10
        for contents in data_parsed.objects.all().iterator():
            try:
                items = unique_variables.objects.get(setid = contents.setid)
            except:
                pass
    timeEnd = datetime.datetime.now()
    print str(timeEnd - timeNow)

However, the problem is that this suddenly consumes hours of time. Since I cannot use select_related, since I'm not going back to sett, but actually moving from sett to unique_variables.

Is there a way to speed up getting items with this behavior?

---

Answer 1

Yes, you can use prefetch_related . I've linked the docs for you.

def something_2():
    timeNow = datetime.datetime.now()
    for i in range (0, N) <---- N is predefined, and no less than 10
        parsed_objects = data_parsed.objects.all().prefetch_related(
            'setid__unique_variables_setid')
        for contents in parsed_objects:
            try:
                items = list(contents.setid.unique_variables_setid.all())
            except:
                pass
    timeEnd = datetime.datetime.now()
    print str(timeEnd - timeNow)