[英]Convert unicode with utf-8 string as content to str
I'm using pyquery to parse a page: 我正在使用pyquery来解析页面:
dom = PyQuery('http://zh.wikipedia.org/w/index.php', {'title': 'CSS', 'printable': 'yes', 'variant': 'zh-cn'})
content = dom('#mw-content-text > p').eq(0).text()
but what I get in content
is a unicode string with utf-8 encoded content: 但我得到的
content
是一个带有utf-8编码内容的unicode字符串:
u'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8...'
how could I convert it to str
without lost the content? 怎么能把它转换成
str
而不丢失内容?
to make it clear: 说清楚:
I want conent == '\\xe5\\xb1\\x82\\xe5\\x8f\\xa0\\xe6\\xa0\\xb7\\xe5\\xbc\\x8f\\xe8\\xa1\\xa8'
我想要
conent == '\\xe5\\xb1\\x82\\xe5\\x8f\\xa0\\xe6\\xa0\\xb7\\xe5\\xbc\\x8f\\xe8\\xa1\\xa8'
not conent == u'\\xe5\\xb1\\x82\\xe5\\x8f\\xa0\\xe6\\xa0\\xb7\\xe5\\xbc\\x8f\\xe8\\xa1\\xa8'
not
conent == u'\\xe5\\xb1\\x82\\xe5\\x8f\\xa0\\xe6\\xa0\\xb7\\xe5\\xbc\\x8f\\xe8\\xa1\\xa8'
If you have a unicode
value with UTF-8 bytes, encode to Latin-1 to preserve the 'bytes': 如果你有一个UTF-8字节的
unicode
值,编码为Latin-1以保留'bytes':
content = content.encode('latin1')
because the Unicode codepoints U+0000 to U+00FF all map one-on-one with the latin-1 encoding; 因为Unicode码点U + 0000到U + 00FF都使用latin-1编码一对一映射; this encoding thus interprets your data as literal bytes.
因此,此编码将您的数据解释为文字字节。
For your example this gives me: 对于你的例子,这给了我:
>>> content = u'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8'
>>> content.encode('latin1')
'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8'
>>> content.encode('latin1').decode('utf8')
u'\u5c42\u53e0\u6837\u5f0f\u8868'
>>> print content.encode('latin1').decode('utf8')
层叠样式表
PyQuery
uses either requests
or urllib
to retrieve the HTML, and in the case of requests
, uses the .text
attribute of the response. PyQuery
使用requests
或urllib
来检索HTML,在requests
的情况下,使用响应的.text
属性。 This auto-decodes the response data based on the encoding set in a Content-Type
header alone, or if that information is not available, uses latin-1
for this (for text responses, but HTML is a text response). 这仅基于
Content-Type
标头中的编码集自动解码响应数据,或者如果该信息不可用,则使用latin-1
(对于文本响应,但HTML是文本响应)。 You can override this by passing in an encoding
argument: 您可以通过传入
encoding
参数来覆盖它:
dom = PyQuery('http://zh.wikipedia.org/w/index.php', encoding='utf8',
{'title': 'CSS', 'printable': 'yes', 'variant': 'zh-cn'})
at which point you'd not have to re-encode at all. 此时你根本不需要重新编码。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.