Python 2.7 + Django 1.7 + PostgreSQL 9.3：尝试将一些文本保存到数据库时出现UnicodeEncodeError。 是什么赋予了？

Question

In my models.py file I have defined a model roughly as follows:

class MyThing(models.Model):
    thing_id     = models.CharField(max_length=20, unique=True, null=False)
    code_snippet = models.TextField(null=True)

I am trying to populate this table with a bunch of data that I am pulling down from a Google API. One piece is a snippet of JavaScript. One of the items being returned from this API has a snippet of JS that is throwing an error.

The error I am getting is:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 213-214: ordinal not in range(128)

The offending bit of text appears in this section:

type="text/javascript" charset="utf-\xad\u20108"></script>

I'm almost certain it is the \\xad, which must be some sort of unicode escape sequence.

I checked my PostgreSQL encoding, and it is using UTF-8.

I also place this line at the top of my models.py, but it didn't make any difference:

from __future__ import unicode_literals

What exactly is going on here? Why can't my unicode-aware database save this string? Also, when did I start using an 'ascii' codec ?

---- EDIT: full stack trace below -----

ERROR/MainProcess] Task apps.r2d_service.tasks.sync_jsnippets_task[e74712bf-2e04-4bed-b08c-f24f9ebb3049] raised unexpected: UnicodeEncodeError('ascii', u'<script type="text/javascript">\n    window._sm_plcmnt = "finyo_5225_5119";\n    var _sm_viewed    = false;\n</script>\n<script src="https://cdn.company412media.com/ng/js/augur.js" type="text/javascript" charset="utf-\xad\u20108"></script>\n<script src=\'https://cdn.company412media.com/ng/pub.js\'></script>\n<script type="text/javascript">\n$$(\'#bib_actions_table tr:nth-child(1) a\').each(function (tab) {\n  tab.observe(\'click\', function (e, el) {\n    if (!_sm_viewed) {\n      SM_Augur.init();\n      _sm_viewed = true;\n    }\n  });\n});\n\n</script>\n', 213, 215, 'ordinal not in range(128)')
Traceback (most recent call last):
  File "/home/vagrant/.virtualenvs/myapp/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/vagrant/.virtualenvs/myapp/local/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/vagrant/myapp/apps/r2d_service/tasks.py", line 25, in sync_jsnippets_task
    sync_jsnippets()
  File "/home/vagrant/myapp/apps/r2d_service/sync.py", line 44, in sync_jsnippets
    myapp_creative.sync(r2d_creative)
  File "/home/vagrant/myapp/apps/r2d_service/models.py", line 244, in sync
    self.save()
  File "/home/vagrant/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/db/models/base.py", line 590, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/vagrant/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/db/models/base.py", line 618, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/home/vagrant/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/db/models/base.py", line 680, in _save_table
    forced_update)
  File "/home/vagrant/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/db/models/base.py", line 724, in _do_update
    return filtered._update(values) > 0
  File "/home/vagrant/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/db/models/query.py", line 600, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/home/vagrant/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1004, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/home/vagrant/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
    cursor.execute(sql, params)
  File "/home/vagrant/.virtualenvs/myapp/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 90, in execute
    logger.debug('(%.3f) %s; args=%s' % (duration, sql, params),
UnicodeEncodeError: 'ascii' codec can't encode characters in position 213-214: ordinal not in range(128)

Answer 1

You need to convert those strings using base64 then you can save it to database.

import base64

base64.b64encode("stings")