通过模型形式将非规范化数据导入Django模型

Question

The Scenario:

I have some data that looks a bit like this:

Person   | Favorite Color | Favorite Fruit
------------------------------------------
Bobby    | RED            | BANANA
Jared    | YELLOW         | RASPBERRY
Milly    | BLACK          | PEACH
Shawn    | ORANGE         | ORANGE

Assume it's in a flatfile, or python dicts, or some other non-sql format.

EDIT: Assume for the sake of argument that I've already got it in a Python structure that looks like this:

data = [
    {"name": "Bobby", "favorite_color": "RED", "favorite_fruit": "BANANA"},
    {"name": "Jared", "favorite_color": "YELLOW", "favorite_fruit": "RASPBERRY"},
    # etc....
 ]

I have django models that look like this:

class Person(models.Model):
    COLORS = (
                 ('R', 'RED'),
                 ('O', 'ORANGE'),
                 ('Y', 'YELLOW'),
                 ('G', 'GREEN'),
                 ('B', 'BLUE'),
                 ('P', 'PURPLE'),
                 ('L', 'BLACK'),
                 ('W', 'WHITE')
              )
    name = CharField(max_length=256)
    favorite_color = CharField(max_length=1, choices=COLORS)
    favorite_fruit = ForeignKey(Fruit)

class Fruit(models.Model):
    name = CharField(max_length=256)
    fructose_content = PositiveIntegerField()

EDIT: Assume that my Fruit model is already populated with all the possible fruits.

The task:

I would like to import my data from the original source into my Django models by using ModelForm s, to take advantage of proper validation and database abstraction.

class PersonForm(forms.ModelForm):
    class Meta:
        model = Person
        fields = '__all__'

Is there a way the ModelForm can translate the denormalized data into data that can be saved in the model? Are ModelForm s the wrong thing to use here?

Answer 1

Try the following code:

insert_data = []

with open('data.txt') as f:
    state = 'HEADER'
    headers = []
    for line in f.readlines():
        if state == 'HEADER':
            headers = [header.lower().strip().replace(' ', '_') for header in line.split('|')]
            state = 'IGNORE'
        elif state == 'IGNORE':
            state = 'DATA'
        elif state == 'DATA':
            data_values = map(str.strip, line.split('|'))
            insert_entry = {}
            for key, data in zip(headers, data_values):
                insert_entry[key] = data
            insert_data.append(insert_entry)

for row in insert_data:
    form = PersonForm(row)

    if form.is_valid():
        form.save()
    else:
        print form.errors()

The first step is to read the file (assuming it is named data.txt), I recommend you using json or some other structured text to avoid typing errors as you can check first if the file is well formatted using well-known libraries.

For this script to work, you will also need a trick for the fields in your forms, I think it will be enough calling the PERSON field as NAME.

On the second step we create the forms instances for every entry we want to insert, validate them and if everything is OK, we save them into the database.

Hope it helps,

Answer 2

I came up with a partial solution, at least for the problem involving the choices. I guess with some tinkering it could work for ForeignKey fields as well.

First, I define a function get_choice_by_name which goes through a choices tuple and looks for a key by value.

Then I subclassed TypedChoiceField and overrode its clean() method to transform the data. This method seems to get called before any validation.

Here's the code:

def get_choice_by_name(name, choices, case_sensitive=False):
    try:
        if name is None:
            return ''
        elif name and not case_sensitive:
            return next(k for k, n in choices
                        if n.lower() == name.lower())
        else:
            return next(k for k, n in choices if n == name)
    except StopIteration:
        raise ValueError(
            "Invalid choice: {}, not found in {}".format(name, choices)
        )

class DenormalizedChoiceField(TypedChoiceField):

    def clean(self, value):
        if not value:
            return self.empty_value
        try:
            value = get_choice_by_name(value, self.choices)
        except ValueError as e:
            raise ValidationError(str(e))

        value = super(DenormalizedChoiceField, self).clean(value)
        return value

My ModelForm now just needs to redefine the fields in question as DenormalizedChoiceField . I need to specify the choices explicitly, though, for some reason it doesn't pick this up from the model if you override the field.

class PersonForm(forms.ModelForm):
    favorite_color = DenormalizedChoiceField(choices=Person.COLORS)
    class Meta:
        model = Person
        fields = '__all__'