UnicodeDecodeError with Django Rest Framework on Model's __str__ field

Question

I'm banging my head on the table regarding this error, and I've scoured the internet but either 1) haven't been able to find the answer or 2) am not knowledgable enough to figure out what's going on.

The tldr summary of my problem:

When I store a non-ASCII character (such as è) in a models.TextField and serialize it with serializers.CharField, I do not encounter and problems. However, if I have a str method on the model that returns that same text field, upon serialization I get the following error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128)

First of all...

• I'm running Python 2.7.10 on OS X.
• I'm using Django==1.11.7 and djangorestframework==3.7.3 .
• My LANG and LC_ALL environment variables are both set to en_US.UTF-8 .
• Lastly, I'm using PostgreSQL 9.5.6 and I verified that my database is encoded in UTF8 .

The details of my problem:

I have a simple model as shown below. It has a name variable that is a TextField as well as a __str__ method that simply returns self.name . I also have defined a testing variable that also simply returns self.name for demonstration purposes.

from __future__ import unicode_literals

from django.db import models
from django.utils.encoding import python_2_unicode_compatible
from django.utils.translation import ugettext_lazy as _


@python_2_unicode_compatible
class Supplier(models.Model):
    name = models.TextField(help_text=_('Store Name'))

    class Meta:
        verbose_name = 'Supplier'
        verbose_name_plural = 'Suppliers'

    def testing(self):
        return self.name

    def __str__(self):
        return self.name

Next, my serializer is defined as follows.

class SupplierSerializer(serializers.ModelSerializer):

    class Meta:
        model = Supplier
        fields = ('name', 'testing', '__str__')

Finally, I have the following viewset:

class SupplierViewSet(viewsets.ModelViewSet):
    queryset = Supplier.objects.all()
    serializer_class = SupplierSerializer

The problem:

When I make a GET request to the view, I get the following error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128)

Observation #1:

This error only seems to occur when I try to serialize __str__ . If I remove __str__ from the serializer (and thus serializer only name and testing ) I don't get any errors. What's odd is that testing returns the same exact thing as __str__ - self.name .

Observation #2: I verified that inside the __str__ method, self.name is type unicode . Inside the testing method, self.name is also type unicode .

Observation #3: The full stack trace is given below.

Traceback (most recent call last):
  File "/Users//github///lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/Users//github///lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/Users//github///lib/python2.7/site-packages/django/core/handlers/base.py", line 217, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users//github///lib/python2.7/site-packages/django/core/handlers/base.py", line 215, in _get_response
    response = response.render()
  File "/Users//github///lib/python2.7/site-packages/django/template/response.py", line 107, in render
    self.content = self.rendered_content
  File "/Users//github///lib/python2.7/site-packages/rest_framework/response.py", line 72, in rendered_content
    ret = renderer.render(self.data, accepted_media_type, context)
  File "/Users//github///lib/python2.7/site-packages/rest_framework/renderers.py", line 105, in render
    allow_nan=not self.strict, separators=separators
  File "/Users//github///lib/python2.7/site-packages/rest_framework/utils/json.py", line 28, in dumps
    return json.dumps(*args, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 210, in encode
    return ''.join(chunks)

It seems to be an error related to the json encoder?

Does anyone with a deeper understanding of django and unicode have an idea what's going on?

Answer 1

While I do not entirely understand the reason why this fixes the error, setting the UNICODE_JSON setting to False fixes the error.

REST_FRAMEWORK = {
    'UNICODE_JSON': False,
}