当 Django 查询集转换为列表时，表情符号的 mySQL Unicode 解码错误

Question

I am on Django 1.11, python 3.6 and mysqlclient 1.4.6. I need to convert a Django queryset into a list. The objects in the queryset have a field with a unicode emoji value. (see: field=u'✅ This is the field value' ) That particular table and field in mySQL are using utf8mb4_unicode_ci encoding.

When I run qs_list = list(qs) , MySQL throws a Unicode error which I think is caused by the emoji in the field when the queryset is evaluated.

File encoding:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

Queryset:

qs = Fake.objects.filter(..)
qs_list = list(qs)

Error:

<class 'UnicodeDecodeError'>
Exception: 'utf-8' codec can't decode byte 0xed in position 11: invalid continuation byte

File "/home/mysite/lib/python3.6/site-packages/django/db/models/query.py", line 250, in __iter__
self._fetch_all()
File "/home/mysite/lib/python3.6/site-packages/django/db/models/query.py", line 1121, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/home/mysite/lib/python3.6/site-packages/django/db/models/query.py", line 53, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File "/home/mysite/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
raise original_exception
File "/home/mysite/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 889, in execute_sql
cursor.execute(sql, params)
File "/home/mysite/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/home/mysite/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 101, in execute
return self.cursor.execute(query, args)
File "/home/mysite/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/home/mysite/lib/python3.6/site-packages/MySQLdb/cursors.py", line 317, in _query
self._post_get_result()
File "/home/mysite/lib/python3.6/site-packages/MySQLdb/cursors.py", line 352, in _post_get_result
self._rows = self._fetch_row(0)
File "/home/mysite/lib/python3.6/site-packages/MySQLdb/cursors.py", line 325, in _fetch_row
return self._result.fetch_row(size, self._fetch_type)

Is there a simple solution or a workaround for this? How can I get the list of objects in the queryset without queryset being evaluated?

Answer 1

Most probably this is related to the encoding and you need to check other types of encoding as suggested in this article

Regarding the list of objects - try iterating on the result to get(for example) each name and store the values in a list:

qs = Fake.objects.filter(..)
qs_list = [x.name for x in qs]

where 'name' is the model field you need to iterate.