将二进制（字节）转换为str，反之亦然

Question

Consider I have a binary data such as in a dataframe 's column :

b'x\\x9c\\xd4\\x14Q\\xd3\\xf7\\x92\\x8b\\x89 \\x01\\xc3)B\\x8c\\x80\\x91#\\x86\\xfb\\xa6\\x9b\\x10\\xce\\x00\\x86p\\x85Cr\\x11\\xd8p\\x84\\xcc\\x12<A\\x17!'

I need an idea how do I convert it to string for each binary column value only, as this dataframe need to be converted to json type and passed as a result of a rest api endpoint.

Here is how I do return dataframe in json format( doing it to dataframe with binary will raise exception):
return json.loads(df.to_json(orient='table'))
Whereas df is a dataframe

I would of course will want to know how to convert back the binary values from string representation of bytes to bytes - binary again.

Answer 1

You need to know the encoding used to create those bytes. The default encoding is platform specific, you check yours:

import sys
sys.getdefaultencoding() # 'utf-8' on macos python 3.7

If you pass a string to pickle.dumps it will encode it using the default encoding, if you want to use another encoding you can encode the string before passing it to pickle.dumps (as an example).

In [2]: pickle.dumps('höy') # will be 'utf-8' encoded by default
Out[2]: b'\x80\x03C\x04h\xc3\xb6yq\x00.'

In [3]: 'höy'.encode('utf-8')
Out[3]: b'h\xc3\xb6y' # compare with the previous output

In [4]: pickle.dumps('höy'.encode('latin1'))
Out[4]: b'\x80\x03C\x03h\xf6yq\x00.'

In [5]: 'höy'.encode('latin1')
Out[5]: b'h\xf6y' # compare with the previous output

Based on the encodings you can decode your strings:

In [1]: 'höy'.encode('utf-8').decode('utf-8')
Out[1]: 'höy'

In [2]: 'höy'.encode('latin-1').decode('latin-1')
Out[2]: 'höy'

Using the wrong encoding will result in failure or wrong result:

In [3]: 'höy'.encode('utf-8').decode('latin-1')
Out[3]: 'höy'

And not every random bunch of bytes is an encoded string:

In [6]: pickle.dumps('höy').decode('utf-8')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-6-2b1872a5aa1a> in <module>
----> 1 pickle.dumps('höy').decode('utf-8')

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte