聚合/优化object.save（）？

Question

I'm working on import feature which allows user to create django database models from selected csv file.

Models are related which each other with foreign keys and many-to-many fields. There is a lot

object.save()

and Object.objects.get(...) in my code which, I suppose, cause it to run so slow.

When an error (for example integrity error) occurs, I need all the changes in database to be rolled back. So I'm using

transaction.atomic 

decorator on my view and it works fine.

The problem is, my import is really slow. Parsing file containing ~2000 lines (which could possibly add about 1000 objects to my database) takes about 3 minutes, which is too long.

Is there a way to make it faster? I've read about

bulk_create

function, but "It does not work with many-to-many relationships.".

If this is important, I'm using postgresql.

EDIT: File structure looks like this:

subject_name
day [A/B] begins_at - ends_at;lecturer_info  

Then multiple lines like:

student_uid;student_info  

Ok, here's the code.

def csv_import(market, csv_file):
    lines = [line.strip().decode('utf-8') for line in csv_file.readlines()]
    lines = [line for line in lines if line]
    pattern = re.compile(r'[0-9]+;.+')   

    week_days = {
        'monday': 0,
        .  
        .
        .
    }

    term, subject, lecturer, student = None, None, None, None

    for number, line in enumerate(lines):
        if not ';' in line:
            subject = Subject(subject_id=number, name=line, market=market)
            subject.save()
        elif not pattern.match(line):
            term_info, lecturer_info = line.split(';')  # term_info - 'day begins_at - ends_at', lecturer_info - lecturer
            term_info = term_info.replace(' - ', ' ').split()
            term = Term(term_id=number, subject=subject, day=week_days[term_info[0]], begin_at=term_info[-2],
                        ends_at=term_info[-1])

            if len(term_info) == 4:
                term.week = term_info[1]

            lecturer_info = lecturer_info.rsplit(' ', 1)
            try:
                lecturer = Lecturer.objects.get(first_name=lecturer_info[0], last_name=lecturer_info[1])
            except Lecturer.DoesNotExist:
                lecturer = Lecturer(first_name=lecturer_info[0], last_name=lecturer_info[1])
                lecturer.save()

            term.lecturer = lecturer

            term.save()
        else:
            gradebook_id, student_info = line.split(';')
            student_info = student_info.rsplit(' ', 1)
            try:
                student = TMUser.objects.get(uid=int(gradebook_id))
            except TMUser.DoesNotExist:
                student = TMUser(uid=int(gradebook_id), username='student'+gradebook_id, first_name=student_info[0],
                                 last_name=student_info[1], password=make_password('passwd'), user_group='user')
                student.save()
            student.terms.add(term)
            student.save()

Answer 1

This is some pseudo code to show you the basic idea of what I meant by caching results:

cache = {}

for number, line in enumerate(lines):
   ...
   elif not pattern.match(line):
      ...
      term = Term(term_id=number, subject=subject, ...)

      lecturer_id = (lecturer_info[0], lecturer_info[1])   #first name and last
      if cache[lecturer_id]:
         #retrieve from cache
         lecturer = cache[lecturer_id]
      else:
         try:
            lecturer = Lecturer.objects.get(first_name= lecturer_id[0], last_name= lecturer_id[1])
         except Lecturer.DoesNotExist:
            lecturer = Lecturer(first_name= lecturer_id[0], last_name= lecturer_id[1])
            lecturer.save()
         #add to cache
         cache[lecturer_id] = lecturer

      term.lecturer = lecturer
      term.save()   

      #etc.