Validating Django model data on bulk_create

Question

I am currently attempting to bulk_create some objects in Django 1.9.2 (Python 3.5.1).

The application will be bulk importing data and I want to validate the data before I insert it into the DB.

The performance using .bulk_create() is, obviously, far superior to .save(), but .full_clean() is killing my performance as well.

view in question:

subscription = get_object_or_404(Subscription, pk=subscription_id)
users = list(range(1,10000))
if users:
    licenses = []
    for u in users:
        license = SubscriptionLicense(resource=subscription.resource,
            external_user_id=int(u), license_expiration_time=None,
            transferable=subscription.license_transferable,
            subscription=subscription
        )

        # Question is in regards to this line
        license.full_clean()

        licenses.append(license)
    SubscriptionLicense.objects.bulk_create(licenses)
    return HttpResponse(status=201)
else:
    raise ValidationError(_('Invalid users provided.'))

Any thoughts on how to validate the objects against model rules without killing performance?

In this instance, I'm running it locally (unittest), and my speed goes from 2.224s to insert 9999 records without validation to 19.592s when I validate using .full_clean()

Answer 1

If you really need this code to be performant, you can manually create a migration with a RunSQL statement specific to your database back-end, where you could add a custom constraint to your database columns.

This is will allow you to use bulk_create() and will raise an IntegrityError if it fails, rather than calling full_clean() and catching the ValidationError .