UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 105: invalid continuation byte

Question

My code is pretty much the following :

import sqlitefts as fts

connection = apsw.Connection('texts.db', flags=apsw.SQLITE_OPEN_READWRITE)
c = connection.cursor()
def do_search(query):
    c.execute('SELECT title, book, author, link, snippet(text_idx) FROM text_idx  WHERE text_idx MATCH 'possumus';') #in this case query='possumus'
    c.fetchall()
--display the results on html--

the query works if i search for more than a single word. But when i do single word searches it throws me an decoding error.

Query that doesn't work are similar to the following:

single word searches like -

SELECT title, book, author, link, snippet(text_idx) 
FROM text_idx  
WHERE text_idx MATCH 'possumus'; 

and OR searches like

SELECT title, book, author, link, snippet(text_idx) 
FROM text_idx  
WHERE text_idx MATCH 'quam OR Galliae';

The stack trace is:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ramcharran/phyllo/search/app.py", line 123, in search
    r = do_search(query)
  File "/home/ramcharran/phyllo/search/app.py", line 90, in do_search
    r1 = c.fetchall()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 105: invalid continuation byte

The query works fine if I do not use the snippet() function in my query.

Answer 1

I think the problem is due to coding. May be tehere are special characters in the text_idx column Try to put:

# Coding: latin1

at the beginning of your file. You must replace latin1 with the encoding that fits your case. I hope I've helped.

Answer 2

There might be special characters in your data. Try encoding your database to utf-8 .