UnicodeEncodeError: &#39;charmap&#39; 编解码器无法对位置 1082-1​​084 中的字符进行编码：字符映射到<undefined>

Question

this is my first time using python, im trying to make CT asset to CSV

this is my code

tsv.py

    def write(self, data):
        """
        Write list to TSV

        :param list data: Decoded list to write on TSV file
        """

        os.makedirs(os.path.dirname("./{0}".format(self.path)), exist_ok=True)
        self.tsv = open(self.path, "w", newline="\r\n")

        self.header = "\t".join(i for i in data[0]) + "\n"
        self.types = "\t".join(i for i in data[1]) + "\n"
        self.rows = "\n".join("\t".join(l) for l in data[2:]) + "\n"

        self.tsv.write(self.header + self.types + self.rows)
        if self.verbose:
            print("\nTSV write to \"{0}\" complete!\n".format(self.path))

And I'm getting the following error:

line 48, in write
    self.tsv.write(self.header + self.types + self.rows)
File "C:\Users\ASUS\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 1082-1084: character maps to <undefined>

What can I do to fix this?

Answer 1

The error message indicates that you are using Python 3 and that it is trying to write text using the Windows code page 1252 encoding. The actual error is that your data contains a character which does not exist in this encoding. I would have expected Python 3 to write UTF-8 but it's not; perhaps you can add encoding="utf-8" to the open call if that is in fact what you want?

self.tsv = open(self.path, "w", newline="\r\n", encoding="utf-8")

If your script works reliably for you, there is no real need to change it; but reinventing CSV/TSV writing is not a good idea -- you are ignoring several corner cases which the existing csv module in the Python standard library already handles.

import csv
import logging

class whatever: ...

    def write(self, data):
        """
        Write list to TSV

        :param list data: Decoded list to write on TSV file
        """

        os.makedirs(os.path.dirname("./{0}".format(self.path)), exist_ok=True)
        with open(self.path, "w", newline="", encoding="utf-8") as tsv:
            writer = csv.writer(tsv, delimiter="\t")
            writer.writerows(data)

        logging.info('TSV write to "%s" complete!', self.path))

Your code also lacks self.tsv.close() which the above code doesn't need because the with handles that transparently.

Additionally, I switched to logging for the informational message. It similarly offers a canned solution which already knows how to write messages to standard error instead of standard output etc (though you might want to look at logging.basicConfig to set up the formatting to your liking -- I usually put logging.basicConfig('%(module)s:%(message)s') for simple command-line utilities).