Send bytes encoded data over JSON

Question

I'm working on an Python RESTful API and I need to send bytes-encoded data (specifically data encrypted using a public RSA key from the package rsa ) over the network, via JSON forms.

Here is what it looks like:

>>> import rsa
>>> pubkey, privkey = rsa.newkeys(512, True) # Create a pair of public/private rsa keys ; 512 is for the example
>>> encStr = rsa.encrypt(b"Test string", pubkey) # Encrypt a bytes object using the public key
>>> encStr
b'r\x10\x03e\xc6*\xa8\xb1\xee\xbd\x18\x0f\x7f\xecz\xcex\xabP~\xb3]\x8f)R\x9b>i\x03\xab-m\x0c\x19\xd7\xa5f$\x07\xc1;X\x0b\xaa2\x99\xa8&\xfc/\x9f\x05!nk\x93%\xc0\xf5\x1d\xf8C\x1fo'

"encStr" is what I need to send, however, I can't tell what encoding it is, and the package documentation doesn't mention it. If you have any idea, please share it:)

>>> encStr.decode("utf-8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec cannot decode byte 0x8e in position 0: invalid start byte
>>> encStr.decode("latin1")
'\x8eM\x96Æ\'zÈZ\x89\x85±\x98Z¯Ûzùæ¯;£zñ8\x9b§Ù\x9dÏ\x8eâ0®\x89ó(*?\x92ªg\x12ôsä\x1d\x96\x19\x82\x19-3\x15SBýh"3òÖß\x91Ô' # This could be it
>>> encStr.decode("latin1").encode("latin1")
b'\x8eM\x96\xc6\'z\xc8Z\x89\x85\xb1\x98Z\xaf\xdbz\xf9\xe6\xaf;\xa3z\xf18\x9b\xa7\xd9\x9d\xcf\x8e\xe20\xae\x89\xf3(*?\x92\xaag\x12\xf4s\xe4\x1d\x96\x19\x82\x19-3\x15SB\xfdh"3\xf2\xd6\xdf\x91\xd4' # Nop, garbage

After manipulating for a while, I found a way to get a proper string using base64.

>>> import base64
>>> b64_encStr = base64.b64encode(encStr)
>>> b64_encStr
b'jk2Wxid6yFqJhbGYWq/bevnmrzujevE4m6fZnc+O4jCuifMoKj+SqmcS9HPkHZYZghktMxVTQv1oIjPy1t+R1A=='
>>> b64_encStr.decode("utf-8")
'jk2Wxid6yFqJhbGYWq/bevnmrzujevE4m6fZnc+O4jCuifMoKj+SqmcS9HPkHZYZghktMxVTQv1oIjPy1t+R1A=='

Now I'd just have to send this, however, I would like to know if there is a more efficient way of doing this (shorter string; less operations, considering the client has to encode and the server decode, etc).

Thanks !

Shawn

Answer 1

Base64 is a relatively efficient method of sending bytes as text (6 bits per character, or 8 bits per byte for normal single byte character encoding). These bytes may have any value, such as found in ciphertext. There are more efficient encodings such as basE91 , but they only provide few advantages for the complexity that they bring.

However, I often see ciphertext being "stringified" while there is no need. Files, HTTP, sockets etc. all handle any byte values well. If you want to use it in a GET request then you should use base64url instead of the normal base 64 encoding. Often developers encode strings needlessly so that the values can be easily seen in traces and such, but in that case only the trace printout itself needs to be encoded.

Note that I'd advise you to use OAEP padding rather than PKCS#1 and a key size of at least 3072 bits, especially if you want to encrypt data that is transported rather than encrypted "in place".