Quickest way to iterate through django database records and store in python dictionary

Question

I am using Django ORM to query two models, each model return huge amount of data( 1500 records/model) and then I iterate through the records and store it in the python dictionary. This takes a very long time for the view to execute and as a result the user is just waiting for the page to load while all this processing happens in the views. Is there any way by which I could make this process fast?

meter = ostk_vm_tenant_quota_stats.objects.filter(cluster=site, collected_at=time.strftime("%Y-%m-%d"))
records = []
for record in meter:
    record_dict = {}
    record_dict['cluster'] = record.cluster
    record_dict['tenant'] = record.tenant
    record_dict['instances_limit'] = record.instances_limit
    record_dict['instances_used'] = record.instances_used
    record_dict['vcpu_limit'] = record.vcpu_limit
    record_dict['vcpu_used'] = record.vcpu_used
    record_dict['memory_limit'] = record.memory_limit
    record_dict['memory_used'] = record.memory_used
    record_dict['disk_limit'] = record.disk_limit
    record_dict['disk_used'] = record.disk_used
    records.append(record_dict)
return render_to_response('tabs1.html', {'data': records})

I do the same thing for other model. "meter" has huge amount of records which I am iterating through to store in the dictionary. Can I make this process faster?

Answer 1

Since (as you mentioned in the comments) record.cluster is a foreign key, every time you access it django is performing another db lookup. This means you are hitting the db for every iteration of the loop. Check the docs on select_related for a more info.

You can either use select_related to prefetch the related clusters, or use record.cluster_id if you only need the primary key value.

# using select related
meter = ostk_vm_tenant_quota_stats.objects \
                                  .select_related('cluster') \
                                  .filter(cluster=site, collected_at=time.strftime("%Y-%m-%d"))

# or only use the pk value in the result dict
record_dict['cluster'] = record.cluster_id

Answer 2

meter = ostk_vm_tenant_quota_stats.objects.filter(cluster=site,
            collected_at=time.strftime("%Y-%m-%d")).values()

will give you a ValueQuerySet, which is basically a Python list of dictionaries. Each dictionary will be all of the fields for each item that matches your query; basically exactly the same as what you have above.