Sqlite database non-ascii character error with python SQLalchemy library

Question

When i use sqlite3 database with sqlalchemy library, i got this error

sqlalchemy.exc.ProgrammingError: (ProgrammingError) 
You must not use 8-bit bytestrings unless you use a text_factory that can
interpret 8-bit bytestrings (like text_factory = str). 
It is highly recommended that you instead just switch your application 
to Unicode strings. 
u'INSERT INTO model_pair (user, password) VALUES (?, ?)' ('\xebE\xc2\xe4.\t312874@gg.com', '123456')

and here is some test data:

 呆呆 3098920@gg.com        11111                                                             
  �言    9707996@gg.com        11111 
wwwj55572@gg?€?       11111

I have configured database encoding as utf-8 or gbk but neither success when insert, i try str.decode('gbk'), it will stuck on char like € and get error like above.

anyone tell me how get around this error ?

Answer 1

try to change '\\xebE\\xc2\\xe4.\\t312874@gg.com' to u'\\xebE\\xc2\\xe4.\\t312874@gg.com'

also, you could try to do '\\xebE\\xc2\\xe4.\\t312874@gg.com'.decode("utf-8") , but it gives an error "invalid continuation byte", perhaps your string is not valid utf-8 after all?

btw, do mention is you are running python 2.x or 3.x, there is a difference.