从JSON文件在Django中进行自定义迁移？

Question

I'm trying to import some initial data into a database in Django from a JSON file and can't for the life of me figure out how to do it in a custom migration. My first function returns dictionaries with fields that match up with the model Mineral in my database. The first two lines of the second function are taken from the Django 1.11 docs on custom migrations, and the rest is just supposed to loop through the JSON file, make dictionaries with the first function, and then create() them, with the keyword arguments coming from the dictionary. But when I try to run it I get django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block. django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

Right now my custom migration file looks like this:

from __future__ import unicode_literals

from django.db import migrations, IntegrityError

import json


def make_mineral_dict(mineral):
    """Make a dictionary out of a mineral object from JSON file"""
    fields = {
        'name': None,
        'image filename': None,
        'image caption': None,
        'category': None,
        'formula': None,
        'strunz classification': None,
        'crystal system': None,
        'unit cell': None,
        'color': None,
        'crystal symmetry': None,
        'cleavage': None,
        'mohs scale hardness': None,
        'luster': None,
        'streak': None,
        'diaphaneity': None,
        'optical properties': None,
        'refractive index': None,
        'crystal habit': None,
        'specific gravity': None,
        'group': None,
    }

    for key, value in mineral.items():
        fields[key] = value
    return fields


def load_data(apps, schema_editor):
    Mineral = apps.get_model("minerals", "Mineral")
    db_alias = schema_editor.connection.alias
    with open('minerals.json', encoding='utf-8') as file:
        minerals = json.load(file)
        for mineral in minerals:
            try:
                fields = make_mineral_dict(mineral)
                Mineral.objects.using(db_alias).create(
                    name=fields['name'],
                    image_filename=fields['image filename'],
                    image_caption=fields['image caption'],
                    category=fields['category'],
                    formula=fields['formula'],
                    strunz_classification=fields['strunz classification'],
                    crystal_system=fields['crystal system'],
                    unit_cell=fields['unit cell'],
                    color=fields['color'],
                    crystal_symmetry=fields['crystal symmetry'],
                    cleavage=fields['cleavage'],
                    mohs_scale_hardness=fields['mohs scale hardness'],
                    luster=fields['luster'],
                    streak=fields['streak'],
                    diaphaneity=fields['diaphaneity'],
                    optical_properties=fields['optical properties'],
                    refractive_index=fields['refractive index'],
                    crystal_habit=fields['crystal habit'],
                    specific_gravity=fields['specific gravity'],
                    group=fields['group']
                )
            except IntegrityError:
                continue



class Migration(migrations.Migration):

    dependencies = [
        ('minerals', '0002_mineral_group'),
    ]

    operations = [
        migrations.RunPython(load_data),
    ]

Any help is greatly appreciated.

Note: Yes, I know I could cut out around 40 lines of code by using **kwargs , but not exactly sure how to do that (I'm still very new); any suggestions on that front would be appreciated as well.

Answer 1

From my understanding, which I would say is fairly limited, a transaction in Django is composed of "atomic parts". Basically the transaction as a whole can be divided up into all the steps for that transaction. Which lets you roll back those steps if 1 fails along the way.

By default, Sqlite and Postgres use transactions. Django Migration by default has atomic = True .

There are a few ways to solve the issue surrounding this.

1. Setting the Migration class atomic = False this would effectively run migration with no transactions.

   • as seen here Django non-atomic

2. Wrapping the "transaction attempt" in a context manager with transaction.atomic()

3. Creating a function with the "transaction attempt" and wrapping that function in a decorator @transaction.atomic

2 and 3 as seen here Django Transactions

I am partial to the context manager solution and keeping transactions in-tact. Because for databases that dont use transactions the docs say:

The atomic attribute doesn't have an effect on databases that don't support DDL transactions (eg MySQL, Oracle).

So for your code example I would throw in the import for transaction. from django.db import transaction

from django.db import transaction
# imports at the top...

def load_data(apps, schema_editor):
    Mineral = apps.get_model("minerals", "Mineral")
    db_alias = schema_editor.connection.alias

    with open('minerals.json', encoding='utf-8') as file:
        minerals = json.load(file)
        for mineral in minerals:
            try:
                fields = make_mineral_dict(mineral)

                # Wrapping this line around...
                with transaction.atomic():
                    Mineral.objects.using(db_alias).create(
                        name=fields['name'],
                        image_filename=fields['image filename'],
                        image_caption=fields['image caption'],
                        category=fields['category'],
                        formula=fields['formula'],
                        strunz_classification=fields['strunz classification'],
                        crystal_system=fields['crystal system'],
                        unit_cell=fields['unit cell'],
                        color=fields['color'],
                        crystal_symmetry=fields['crystal symmetry'],
                        cleavage=fields['cleavage'],
                        mohs_scale_hardness=fields['mohs scale hardness'],
                        luster=fields['luster'],
                        streak=fields['streak'],
                        diaphaneity=fields['diaphaneity'],
                        optical_properties=fields['optical properties'],
                        refractive_index=fields['refractive index'],
                        crystal_habit=fields['crystal habit'],
                        specific_gravity=fields['specific gravity'],
                        group=fields['group']
                    )
            except IntegrityError:
                continue