How do you access a Django JSONField's keys using `django.models.F` or similar method during a query?

Question

Is it possible to extract a sub-key from a JSONField field and annotate the Queryset with its value? I'm trying to extract the value within the query rather than post-processing in the Python code.

Model architecture is:

• Django 1.10
• Model has a django.contrib.postgres.fields.JSONField called data` to store an API response. This example is Twitter.
• Other fields are profile_id and screen_name . The rest of the data lives within the data field so it can be queried ad-hoc.

I thought I'd be able to combine annotate and django.models.F but I'm getting the following error:

> models.TwitterUser.objects.annotate(foll_count=F("data__followers_count"))

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/query.py", line 914, in annotate
    clone.query.add_annotation(annotation, alias, is_summary=False)
  File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/sql/query.py", line 971, in add_annotation
    summarize=is_summary)
  File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/expressions.py", line 463, in resolve_expression
    return query.resolve_ref(self.name, allow_joins, reuse, summarize)
  File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1462, in resolve_ref
    self.get_initial_alias(), reuse)
  File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1402, in setup_joins
    names, opts, allow_many, fail_on_missing=True)
  File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1370, in names_to_path
    " not permitted." % (names[pos + 1], name))
django.core.exceptions.FieldError: Cannot resolve keyword 'followers_count' into field. Join on 'data' not permitted.

This isn't explicitly documented anywhere so I'm attempting to reverse engineer it using the double underscores used elsewhere in Django. I've separately tried accessing the key as if it was a native Python dict ( F("data")[followers_count"] ) but that didn't work either.

Any direct answers or pointers towards other areas would be appreciated.

Answer 1

I couldn't use F() at the time of writing so had to fallback to a RawSQL call to access the field.

Based on the prior work of How to aggregate (min/max etc.) over Django JSONField data? and José San Gil on his blog.

qs = models.TwitterUser.objects.annotate(followers=RawSQL(
        # The Postgres SQL query to access a JSON object field.
        # `data` is the name of the JSONField in my schema.
        # More: https://www.postgresql.org/docs/9.3/static/functions-json.html
        "((data->%s))",  
        # The parameter to insert into the query and replace '%s'
        # This could be a variable
        # and the query adapted into a reusable universal function.
        ("followers_count",)  # In my example, this is the JSON sub-key I'm after
        )
    )

Note: I've bracketed and indented to hopefully aid comprehension.

You can also add a .filter(data__has_key="insert_key_here") before the annotate if you only want to return items that contain the field in question. This is a nice native method for JSON within the ORM and hopefully in time we'll have similar ways of accessing JSON sub-fields directly through the ORM.