Python MySQLdb上传UnicodeEncodeError

Question

I have a problem where I can upload CSV files to MySQL, but then something happens and I get an encoding error. Can some one please review my code and tell what is wrong? I'm new to enconding.

The following snippet is how I write the CSV files that will be uploaded, the data is extracted from an MDB file using the MDN tools (mdb-export):

    tableIndex  = 1
    for tName in tableNames:
        fileName = os.path.join(csvPath, os.path.basename(mdb).split('.')[0] + '_' + tName + '.csv')

        try:
            p = subprocess.Popen(["mdb-export", "-H", mdb, tName], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            tableContent, error = p.communicate()

            if(p.returncode != 0):
                _logger.error('[%3d] Export Subprocess %d %s' % (tID, p.returncode, tableContent))
                SendMdbError(tID, mdb, _logger, 'ALERT: Export Subprocess')
                return(['', False])
            if(error):
                _logger.error('[%3d] Export Communicate %d %s' % (tID, p.returncode, error.strip()))
                SendMdbError(tID, mdb, _logger, 'ALERT: Export Communicate')
                return(['', False])

        except Exception as ex:
            _logger.exception('[%3d] Export Error' % tID)
            SendMdbError(tID, mdb, _logger, 'ALERT: Export Exception')
            return(['', False])
        except:
            _logger.exception('[%3d] Export Unexpected' % tID)
            SendMdbError(tID, mdb, _logger, 'ALERT: Export Unexpected')
            return(['', False])

        # If no data, no need for corresponding SQL
        if(len(tableContent) == 0):
            emptyTables.append(tName)

        # If data exists, dump data
        else:
            # Add the 'DriveTest' to the data to upload
            tableContent = tableContent.split('\n')

            tableContent = [dt + ',' + line for line in tableContent if(line)]
            tableContent = '\n'.join(tableContent)

            try:
                with open(fileName, 'wb') as f:
                    f.write(tableContent)

                    if(_VERBOSITY):
                        _logger.debug('[%3d] %3d - Write CSV SIZE[%8d] FILE: %s' %(tID, tableIndex, len(tableContent.split('\n')), fileName))
                        tableIndex += 1

            except IOError as err:
                _logger.exception('[%3d] Write IOError: %s' % (tID, str(err)))
                SendMdbError(tID, mdb, _logger, 'ALERT: Write IOError')
                return(['', False])
            except Exception as ex:
                _logger.exception('[%3d] Write Exception' % tID)
                SendMdbError(tID, mdb, _logger, 'ALERT: Write Exception')
                return(['', False])
            except:
                _logger.exception('[%3d] Write Unexpected: %s' % tID)
                SendMdbError(tID, mdb, _logger, 'ALERT: Write Unexpected')
                return(['', False])

The following is where I upload the CSV file, and here is where I get the error:

    # Upload the data
    tableIndex = 0
    for table in tableDDL:
        try:

            with warnings.catch_warnings(record=True) as war:

                _logger.info('[%3d] %3d Going up... %s' %(tID, tableIndex+1, os.path.basename(mdb).split('.')[0] + '_' + table))

                _sqlLock[tableIndex].acquire()
                #self.cursor.execute(tableDDL[table])
                self.cursor.execute(tableULD[table])
                self.conn.commit()
                _sqlLock[tableIndex].release()

                if(war):
                    #if(_VERBOSITY): print('[%3d] %3d WARNINGS[%3d] %s' % (tID, tableIndex+1, len(war), os.path.basename(mdb).split('.')[0] + '_' + table))
                    _logger.warning('[%3d] %3d WARNINGS[%3d] %s' % (tID, tableIndex+1, len(war), os.path.basename(mdb).split('.')[0] + '_' + table))
                    for w in war:
                        _logger.warning('[%3d] %s' % (tID, w.message))

                #if(_VERBOSITY): print('[%3d] %3d Uploaded %s' % (tID, tableIndex+1, os.path.basename(mdb).split('.')[0] + '_' + table))
                _logger.info('[%3d] %3d Uploaded %s' % (tID, tableIndex+1, os.path.basename(mdb).split('.')[0] + '_' + table))
                tableIndex += 1

                # Remove the uploaded CSV file
                try:
                    os.remove(csvFiles[table]+'.csv')
                    _logger.info('[%3d] Removed CVS file: %s' % (tID, csvFiles[table]+'.csv'))
                except OSError:
                    pass

        except (MySQLdb.InternalError, MySQLdb.NotSupportedError) as err:
            _logger.error('[%3d] %3d Internal: %s %s' % (tID, tableIndex+1, err, sys.exc_info()[0]))
            self.conn.rollback()
            self.Disconnect(tID, _logger, _VERBOSITY, _DEBUG)
            return(False)
        except MySQLdb.OperationalError as err:
            _logger.error('[%3d] %3d OperationalError: %s' % (tID, tableIndex+1, sys.exc_info()[0]))
            _logger.error(err)
            self.conn.rollback()
            self.Disconnect(tID, _logger, _VERBOSITY, _DEBUG)
            return(False)
        except MySQLdb.ProgrammingError as err:
            _logger.error('[%3d] %3d ProgrammingError: %s' % (tID, tableIndex+1, sys.exc_info()[0]))
            _logger.error(err)
            self.conn.rollback()
            self.Disconnect(tID, _logger, _VERBOSITY, _DEBUG)
            return(False)
        except MySQLdb.Error as err:
            _logger.error('[%3d] %3d QUERY: %s %s' % (tID, tableIndex+1, err, sys.exc_info()[0]))
            self.conn.rollback()
            self.Disconnect(tID, _logger, _VERBOSITY, _DEBUG)
            return(False)
        except Exception as err:
            _logger.error('[%3d] %3d Exception: %s %s' % (tID, tableIndex+1, err, sys.exc_info()[0]))
            #self.conn.rollback()
            #self.Disconnect(tID, _logger, _VERBOSITY, _DEBUG)
            #return(False)
            pass
        except:
            _logger.error('[%3d] %3d Other: %s' % (tID, tableIndex+1, sys.exc_info()[0]))
            self.conn.rollback()
            self.Disconnect(tID, _logger, _VERBOSITY, _DEBUG)
            return(False)

The error I get is the following:

2015-06-13 19:42:21,743 __main__ -    ERROR - [  1]   1 Exception: 'ascii' codec can't encode character u'\xb4' in position 40: ordinal not in range(128) <type 'exceptions.UnicodeEncodeError'>
2015-06-13 19:42:30,962 __main__ -    ERROR - [  1]   1 Exception: 'ascii' codec can't encode character u'\xb4' in position 27: ordinal not in range(128) <type 'exceptions.UnicodeEncodeError'>

I noticed that the given data gets uploaded, but not sure if all rows are uploaded.

Thanks!

Answer 1

Try before putting csv into DB s.decode('UTF-8') and after getting it out of the DB s.encode('UTF-8')

I did it for SQLite and it worked OK.

Answer 2

Getting this to work should not be too difficult, but you have to understand what you're doing. Don't just try all possible combinations of s.encode("UTF-8").decode("UTF-8") and stuff like that.

First, understand the difference between a string and bytes . See https://docs.python.org/3/howto/unicode.html . You can encode a string to bytes: bytes = text.encode("UTF-8") , and you can decode bytes to a string: text = bytes.decode("UTF-8")

Second since a CSV file is a text file, you should open the CSV file in text mode. open(fileName, 'w', encoding="utf-8") . There's no need to encode or decode text in your code when writing the file.

Third, it is perfectly OK to write Unicode text to a TEXT field. No need for BINARYs or BLOBs. But make sure your database has a collation setting that can deal with it, usually that would be one of the utf-8 collations. Then to put Unicode in your database, use python strings and don't decode them to bytes.

Answer 3

The error message implies that the column definition in MySQL is CHARACTER SET ascii ; is that correct?

B4 sounds like the latin1 (not utf8) encoding for ´ , which could be coming from a Microsoft Word document in a context such as it´s .

So, even changing the column to be CHARACTER SET utf8 won't fix the problem.

BINARY and BLOB are essentially the same type of field -- any byte is allowed. VARCHAR and TEXT validate the bytes during INSERT to make sure they match the CHARACTER SET .