python unicode csv export using pyramid

Question

I'm trying to export mongodb that has non ascii characters into csv format. Right now, I'm dabbling with pyramid and using pyramid.response.

from pyramid.response import Response
from mycart.Member import Member

@view_config(context="mycart:resources.Member", name='', request_method="POST", permission = 'admin')
def member_export( context, request):
     filename = 'member-'+time.strftime("%Y%m%d%H%M%S")+".csv"
     download_path = os.getcwd() + '/MyCart/mycart/static/downloads/'+filename

     member = Members(request)

     my_list = [['First Name,Last Name']]

     record = member.get_all_member( )             
     for r in record:

         mystr = [ r['fname'],  r['lname']]

         my_list.append(mystr)

     with open(download_path, 'wb') as f:
         fileWriter = csv.writer(f, delimiter=',',quotechar='|', quoting=csv.QUOTE_MINIMAL)
         for l in my_list:
             print(l)
             fileWriter.writerow(l)

     size = os.path.getsize(download_path)
     response = Response(content_type='application/force-download', content_disposition='attachment; filename=' + filename)
     response.app_iter = open(download_path , 'rb')
     response.content_length = size

     return response

In mongoDB, first name is showing 王 , when I'm using print, it too is showing 王 . However, when I used excel to open it up, it shows random stuff - ç¾…

However, when I tried to view it in shell

$ more member-20130227141550.csv

It managed to display the non ascii character correctly.

How should I rectify this problem?

Answer 1

I'm not a Windows guy, so I am not sure whether the problem may be with your code or with excel just not handling non-ascii characters nicely. But I have noticed that you are writing your file with python csv module, which is notorious for headaches with unicode .

Other users have reported success with using unicodecsv as a replacement for the csv module. Perhaps you could try dropping in this module as a csv writer and see if your problem magically goes away.