使用python编写csv时出错，可能是unicode / encode / decode问题

Question

I've been trying to find answers elsewhere but either I do not understand the explaination or the resolution does not work for my case.

So for this case:
1. the output character is Chinese
2. the reading part works perfectly fine, just the writing malfunction
3. I'm using Python 2.7.13

Please help!

BTW, I'm pretty new to python, so if you located anything that could be improved by using any better practices please point them out! I would really appreciate it!

Thank you!

Here's the code:

# -*- coding: utf-8 -*-
import csv
import urllib2
from bs4 import BeautifulSoup
import socket
import httplib
# import sys  <= this did not work
# reload(sys)
# sys.setdefaultencoding('utf-8')

with open('/users/Rachael/Desktop/BDnodes.csv', 'r') as readcsv, 
open("/users/Rachael/Desktop/CheckTitle.csv", 'wb') as writecsv:
    writer = csv.writer(writecsv)
    for row in readcsv.readlines():
        opener = urllib2.build_opener()
        opener.addheaders = [('User-Agent',
                          'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1941.0 Safari/537.36')]
        urllib2.install_opener(opener)
        openpage = urllib2.urlopen(row).read()
        soup = BeautifulSoup(openpage, "lxml")
        # print "page results:"
        for child in soup.findAll("h3", {"class": "t"}):
            try:
                geturls = child.a.get('href')
                # print urllib2.urlopen(geturls).geturl()
                url_result = urllib2.urlopen(geturls).geturl()
                # print url_result
                try:
                    openitem = urllib2.urlopen(url_result).read()
                    gettitle = BeautifulSoup(openitem, 'lxml')
                    url_title = gettitle.title.text
                except urllib2.HTTPError:
                    url_title = 'passed http error'
                    pass
                except urllib2.URLError:
                    url_title = 'passed url error'
                    pass
                except socket.timeout:
                    url_title = 'passed timeout'
                    pass
                except httplib.BadStatusLine:
                   url_title = 'passed badstatus'
                    pass
                except:
                    url_title = 'unknown'
                    pass
            except urllib2.HTTPError as e:
                pass
            except urllib2.URLError:
                pass
            except socket.timeout:
                pass
            except httplib.BadStatusLine:
                pass
            writer.writerow([url_result, url_title])
            # writer.writerow([url_result, url_title.encode('utf-8')]) did not work either, even tried with 'utf-16'
writecsv.close()

The error was:

C:\Python27\python.exe C:/Users/Rachael/PycharmProjects/untitled1/OpenNGet.py
Traceback (most recent call last):
  File "C:/Users/Rachael/PycharmProjects/untitled1/OpenNGet.py", line 55, in <module>
    writer.writerow([url_result, url_title])
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

Process finished with exit code 1

Answer 1

You can pass encoding parameter in open function.

import codecs
codecs.open("/users/Rachael/Desktop/CheckTitle.csv", 'wb', encoding='utf-8') as writecsv

Answer 2

Could it be that your original solution is correct, but that the problem is in the 'result' variable instead of in the title?

Try something like

writer.writerow([url_result.encode('utf-8'), url_title.encode('utf-8')])