How can I use keyword arguments when subclassing django's model FileField

Question

I'm trying to subclass Django's models.FileField but somewhere, the instantiation is going very wrong.

Here's my code:

class DatafileObjectField(models.FileField):

    def __init__(self, source_key=None, **kwargs):
        print('SOURCE KEY IS:', source_key)
        source = settings.DATA_SOURCES[source_key]
        kwargs["storage"] = import_string(source["storage"])(**source["storage_settings"])
        super().__init__(**kwargs)


class MyModel(models.Model):

    # THIS IS THE ONLY REFERENCE IN MY ENTIRE CODEBASE TO THIS FIELD
    # Yet somehow the field is instantiated by django without the source_key argument
    file = DatafileObjectField(source_key='my-data-source', help_text="Upload a data file containing mast timeseries")

When I do:

python mange.py makemigrations

That print statement is issued twice:

SOURCE KEY IS: my-data-source
<output from checks>
SOURCE KEY IS: None

Followed by the inevitable error, because the second instantiation of the subclassed DatafileObjectField doesn't receive the source_key argument.

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/makemigrations.py", line 142, in handle
    ProjectState.from_apps(apps),
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/state.py", line 220, in from_apps
    model_state = ModelState.from_model(model)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/state.py", line 409, in from_model
    fields.append((name, field.clone()))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/__init__.py", line 514, in clone
    return self.__class__(*args, **kwargs)
  File "/app/backend/datalake/models/mast_timeseries.py", line 52, in __init__
    source = settings.DATA_SOURCES[source_key]
KeyError: None

What on earth is django doing? Why isn't source_key always passed?

Answer 1

A Model / model field needs to be serializable to be able to use it in the migrations system. Django uses the deconstruct method on classes to make them serializable. This method according to the documentation :

Returns a 4-tuple with enough information to recreate the field:

1. The name of the field on the model.
2. The import path of the field (eg "django.db.models.IntegerField"). This should be the most portable version, so less specific may be better.
3. A list of positional arguments.
4. A dict of keyword arguments.

Hence you need to override this method and pass your own kwarg to it so that the field can be serialized:

class DatafileObjectField(models.FileField):

    def __init__(self, source_key=None, **kwargs):
        print('SOURCE KEY IS:', source_key)
        source = settings.DATA_SOURCES[source_key]
        kwargs["storage"] = import_string(source["storage"])(**source["storage_settings"])
        super().__init__(**kwargs)
    
    def deconstruct(self):
        name, path, args, kwargs = super().deconstruct()
        kwargs['source_key'] = get_source_key_from_storage(self.storage) # Somehow get source key here?
        return name, path, args, kwargs

Note : I use some non-existent function get_source_key_from_storage since you haven't set it on the instance of the field. You can get the source key somehow or set it on the instance and use that here.

Answer 2

OK, figured it out. I'd seen the mention of the deconstruct method in django docs but not fully absorbed its purpose.

Basically, the Field is instantiated (correctly, that's the first print output in the question) then its deconstruct method is used by makemigrations to determine what pieces of information need to be saved in the migration .

It tells the migration what's important to actually store.

So in my case I needed to add the storage_key kwarg into the values returned from the deconstruct() method (which meant saving it on the model).

In this particular use case, because I'm determining what storage class to use on instantiation, I don't also need to serialise the storage kwarg, so I delete that.

Working solution:

class DatafileObjectField(models.FileField):

    def __init__(self, source_key=None, **kwargs):
        self.source_key = source_key
        source = settings.OCTUE_DATA_SOURCES[source_key]
        kwargs["storage"] = import_string(source["storage"])(**source["storage_settings"])
        super().__init__(**kwargs)

    def deconstruct(self):
        name, path, args, kwargs = super().deconstruct()
        # Make sure the migration knows that this settings key is required for instantiation
        kwargs["source_key"] = self.source_key
        # Remove this key, which is no longer required (calculated during __init__)
        kwargs.pop("storage")
        return name, path, args, kwargs