Scrapy-如何在我的输出上转换unicode？

Question

I'm scraping a website and the titles have latin accents Ex: É, não, etc...

This is my code:

    for tank in response.xpath('//html/body/div/div[4]/div/div/div/table[1]/tr/td/div'):
        item = VapeItem()
        item["title"] = tank.xpath("h3/a/text()").extract()

And the Json Output example:

{"title": "HALO CAF\u00c9 MOCHA"},

Question is: How do I convert this so it shows up like this?

 {"title": "HALO CAFÉ MOCHA"},

I've tried encode("utf8") without success.

Answer 1

You probably need to just print it?

>>> print json.loads(txt)['title']

HALO CAFÉ MOCHA

Writing to a file works just as well, don't really see the problem here.

>>> parsed_data = json.loads('{"title": "HALO CAF\u00c9 MOCHA"}')
>>> with open('foo.txt', 'w') as fin:
...   fin.write(parsed_data['title'].encode('utf-8'))
... 

Answer 2

You've got it backwards. You need to decode as utf-8 (to convert from bytes-like str data to unicode ).

But that's not the real problem: json dump ensures ASCII compatible output by default (using escapes) to avoid problems with protocols that only handle ASCII (or can't rely on a specific encoding besides "ASCII compatible").

Pass ensure_ascii=False to the dump / dumps call to allow it to output non-ASCII. Note the warnings on the docs; this can make some calls return str , others unicode , which may cause problems (on Py3, the issues aren't there; it's always str ).