无法打开包含西里尔字母的Python编码URL

Question

I have the following URL "mysite.com/\Т\е\к\с\т \н\а \к\и\р\и\л\и\ц\а" ("mysite.com/Текст на кирилица"). I want to open this URL using browser.open(link) where browser is

$CHandler = urllib2.HTTPCookieProcessor(cookielib.CookieJar())
browser = urllib2.build_opener(CHandler)
user_agent = '  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17'
browser.addheaders = [('User-agent', user_agent )]
urllib2.install_opener(browser)

However I get the error:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 12-17: ordinal not in range(128)"

I get this URL from JSON.

How can I resolve this?

Answer 1

mysite.com/Текст на кирилица is not a URL:

• because it has omitted the http:// (or other) schema;
• it has spaces in, which aren't valid;
• because URIs can't contain non-ASCII characters. Only IRIs can, and urllib2 doesn't support them.

So you will need to fix the brokennesses, %-encoding out of band characters (like space -> %20 ), add the schema if missing, and then convert IRI to URI. To do this conversion you will need to encode the hostname part of the address using the IDN algorithm (Python: s.encode('idna') ), then encode any non-ASCII characters in other parts of the address using UTF-8 then %-encoding.

What you want to end up with is:

http://mysite.com/%D0%A2%D0%B5%D0%BA%D1%81%D1%82%20%D0%BD%D0%B0%20%D0%BA%D0%B8%D1%80%D0%B8%D0%BB%D0%B8%D1%86%D0%B0

which is a valid URI accepted by urllib2 , but also displays as http://mysite.com/Текст на кирилица in the browser's address bar when you follow it.

There are lots of functions about that implement IRI-to-URI (most Python web frameworks have something like it, for one). If you want to go the whole hog on correcting and normalising suspect incoming URLs, there's also this .