Python 以 utf-8 或 cp1252 格式导入 .csv 文件

Question

I asked a question a while back about dealing with import of .csv files with special characters. At the time I was interested in solving the 90% case, but now I'm back for the last 10%.

It's mostly the same setup as before:

1. Many input files
2. All .csv
3. New: Now I want to preserve special characters in some inputs. However, I don't have control over the format of all of my input files, so I have a mix of files that I need to process. My attempt at the solution was to pass a keyword argument when I want to do a different encoding format.

Here is the code:

import csv
import unicodecsv
#<Lots of other declarations and initialization>

def _csv_dict(self, file,index_field, ScrubMe, **kwargs):

#some irrelevant initialization stuff here.

    if 'formatting' in kwargs:
        formatting = kwargs['formatting']
    else:
        formatting =  None #cp1252 is OS default

    with open(file, encoding=formatting, errors='ignore') as f: #newline = '',
        if formatting == None:
            reader = csv.DictReader(f, dialect = 'excel')
        else: #assume for now UTF-8 is the only other supported format
            reader = unicodecsv.DictReader(f, dialect = csv.excel)

        for line in reader:
            <do some stuff - it's mostly building dictionaries, but I
generally edit the data to only keep the stuff I care about and do a little
data transformation to standard formats >

The result of the above is that if I pass an Excel file saved as a .CSV in native codec, the import works. However, the unicodecsv file with call including formatting='utf-8' keyword arg crashes

The error message suggests that I'm passing the wrong type of object somewhere along the line. This happens the first time I attempt to read a line out of the UTF-8 file

File 
"C:\Users\<me>\AppData\Local\Programs\Python\Python37\lib\site-
packages\unicodecsv\py3.py", line 51, in <genexpr>
f = (bs.decode(encoding, errors=errors) for bs in f)
AttributeError: 'str' object has no attribute 'decode'

From what I have read, UTF-8 is actually tab-separated instead of comma-separated, but I "think" it's supposed to work the same way.

I feel like I've probably messed up something pretty simple, but I've killed enough time looking that it seems appropriate to ask for help. Thanks in advance for any suggestions.

Answer 1

I'm replacing my initial answer because I had multiple things going on and it took me a while to untangle them.

1) @lenz is correct. In Python 3 it is unnecessary to use unicodecsv.DictReader. Part of what confused me is the difference in implementation.

a) For the older unicodecsv.DictReader from Python 2:

kw_args={'errors' : None}
with open(filename, 'rb', **kw_args) as file:
    reader = unicodecsv.DictReader(file, dialect = csv.excel, encoding='utf_8_sig' )

b) For Python 3 csv.DictReader

kw_args={'newline' : '','errors' : None,'encoding' : 'utf_8_sig'}
with open(filename, 'r', **kw_args) as file:
    reader = csv.DictReader(file, dialect = csv.excel )

To summarize the differences

• Mode of file open is now text instead of bytes
• Because of the different open method, the codec can/should be specified in the file open vs. in the DictReader
• newline parameter is also only valid for the file opened as text.

2) Because my UTF-8 file was produced by Excel, it has a utf_16_le style BOM at the top of the file. The only coded that works for this is 'utf_8_sig'.

3) Because my output files are being read downstream by SQL Server, the output codec needs to be 'utf_16_le' or SQL Server doesn't recognize it.

4) Also, because the target is SQL Server, I have to manually insert the BOM at the top of the file.

csvfile.write('\uFEFF') 
writer.writeheader()

If you open the above output file in Excel it will no longer be in columns, but SQL Server (actually SSIS) now knows how to read the file.

5) Just to mess with me a little more, someone had '\\n' in a few of the records. With Excel as source and destination, this was not an issue, but it was for SSIS. My solution:

for r in record_list:
    temp={}
    for k,v in r.items():

        if isinstance(v,str):
            temp[k] = v.replace('\n',' ')
        else:
            temp[k] = v
    writer.writerow(temp)