Python, ascii can't encode byte error

Question

Hi I have an abstract that I download from a website, sometimes it doesn't have special chars and it doesn't give me any problem when I try to store it in the db...

Other times I get an error, Python can't encode byte error blah blah blah...

I use .encode('ascii', 'ignore') in order to avoid those issues but in the end those special characters don't get saved... what should I use to store every character without problems??

Answer 1

As Alex mentioned, you first need to determine how the file is encoded ( response.encoding in scrapy). Assuming UTF-8, you'd simply .encode('utf-8') , or any of Python's other supported encodings .

You also need to make sure that the table in which you are trying to store the data accepts that particular encoding. See this question for examples on how to enable UTF-8 storage. You can also use any of the other character sets that MySQL supports in the same manner.