how to convert UTF-8 code to symbol characters in python

Question

I crawled some webpages using python's urllib.request API and saved the read lines into a new file.

        f = open(docId + ".html", "w+")
        with urllib.request.urlopen('http://stackoverflow.com') as u:
              s = u.read()
              f.write(str(s))

But when I open the saved files, I see many strings such as \\xe2\\x86\\x90, which was originally an arrow symbol in the original page. It seems to be a UTF-8 code of the symbol, but how do I convert the code to the symbol back?

Answer 1

Your code is broken: u.read() returns bytes object. str(bytes_object) returns a string representation of the object (how the bytes literal would look like) -- you don't want it here:

>>> str(b'\xe2\x86\x90')
"b'\\xe2\\x86\\x90'"

Either save the bytes on disk as is:

import urllib.request

urllib.request.urlretrieve('http://stackoverflow.com', 'so.html')

or open the file in binary mode: 'wb' and save it manually:

import shutil
from urllib.request import urlopen

with urlopen('http://stackoverflow.com') as u, open('so.html', 'wb') as file:
    shutil.copyfileobj(u, file)

or convert bytes to Unicode and save them to disk using any encoding you like.

import io
import shutil
from urllib.request import urlopen

with urlopen('http://stackoverflow.com') as u, \
     open('so.html', 'w', encoding='utf-8', newline='') as file, \
     io.TextIOWrapper(u, encoding=u.headers.get_content_charset('utf-8'), newline='') as t:
    shutil.copyfileobj(t, file)

Answer 2

Try:

import urllib2, io

with io.open("test.html", "w", encoding='utf8') as fout:
    s = urllib2.urlopen('http://stackoverflow.com').read()
    s = s.decode('utf8', 'ignore') # or s.decode('utf8', 'replace')
    fout.write(s)

See https://docs.python.org/2/howto/unicode.html