如何通过 python 雪花连接器处理 Unicode 字符问题，同时从雪花读取数据

Question

While reading data from snowflake using python snowflake connector, I am getting the following error:

"InterfaceError: 252005: Failed to convert current row, cause: 'utf-8' codec can't decode byte 0xe1 in position 316: invalid continuation byte"

The string contains non-UTF-8 characters and the snowflake cursor is unable to return the value. How to handle this situation, content is required.

Python version 3.7.6 Snowflake Python connector 5.5.1

Sample Code:

import snowflake.connector 

ctx = snowflake.connector.connect(user='user', password='pwd',account='act',warehouse='wrh', database='db', schema='schema', role = 'role' ) 
cur = ctx.cursor() 
cur.arraysize = 10000 
sql = """select longText from db.schema.table where textId = 1279""" 
cur.execute(sql) 
for element in cur: 
   print(element[0])

Sample Data:

xxxx: xxxxxxxx@xxxxx.xxx Tx: xttxxh@xxxxx.xxx xx: xuxxxxt: xx: xxx00x0x3 : xxxxxxx0x= // Hxvx x xxxx xx-Xxxxx-xxx thxt xxxt xxxx. xxgx xhxx thxt thx xhxxxx (Uxxxxxxxxxxx) -----xxxgxxxx xxxxxgx----- xxxx: xxxx, xxux x xxV (Ux) [xxxxtx:xxux.x.xxxx.xxv@xxxx.xxx] xxxt: xxxxxxxxy, xxvxxxxx xx, x01x 1x:1x xx Tx: xxxxxáx xx xxxáxxx xx: xxxx, xxux x xxV (Ux) xuxxxxt: xx: xxx00x0x3 : xxxxxxx0x= // Hxvx x xxxx xx-Xxxxx-xxx thxt xxxt xxxx. xxgx xhxx thxt thx xhxxxx (Uxxxxxxxxxxx)

Answer 1

Instead of select longText , try select hex_encode(longText) . This will transform whatever strange binary text lives inside that field into a safe to transport string. Then you can decode that back within the safety of Python.

For example, in sql:

select hex_encode('hello')

That returns 68656C6C6F , which you can decode back in Python:

>>> bytes.fromhex('68656C6C6F')
b'hello'

This technique will also give you some insights on what the weird character tripping up your code might be - and what encoding to use.

>>> bytes.fromhex('68656C6C6F')
b'hello'
>>> bytes.fromhex('F09F9988')
b'\xf0\x9f\x99\x88'
>>> bytes.fromhex('F09F9988').decode('utf-8')
'🙈'