UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 65535: unexpected end of data

Question

When I use :

print(bytes(my_string,'utf-8').decode('utf-8'))

everything is okey and I can see my string.

But When I send my_string to another machine, with the following command :

conn.send(bytes(my_string,'utf-8'))

And receive it in another machine, But When I want to decode it, I get the following error:

print(recv_data.decode('utf-8'))

  File "./multiconn-client.py", line 86, in service_connection
    print(recv_data.decode('utf-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 65535: unexpected end of data

And When I print without encoding :

print(recv_data)

I see a text But it it is not in my encoding.

How to solve it?

UPDATE:

My server :

   conn.send(bytes(my_string,'utf-8'))  
   conn.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, sys.getsizeof(my_string))

NOTE: When I print my_string in server everything is OK.

I use selectors module for more than one connection.

In client I use :

def service_connection(key, mask):
sock = key.fileobj
data = key.data
if mask & selectors.EVENT_READ:
    buff = sock.getsockopt(SOL_SOCKET,SO_RCVBUF)
    recv_data = sock.recv(1024 | buff)
    if recv_data:
        buff = sock.getsockopt(SOL_SOCKET,SO_RCVBUF)
        print('In cli getsizeof of recv_data:',sys.getsizeof(recv_data))
        recv_data += recv_data
        data.recv_total += len(recv_data)
        print(recv_data.decode('utf-8')) #######error line is here.

Mow I get the folllowing error only for decode function:

Traceback (most recent call last):
  File "./multiconn-client.py", line 122, in <module>
    service_connection(key, mask)
  File "./multiconn-client.py", line 99, in service_connection
    print(recv_data.decode('utf-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 65535: invalid continuation byte

NOTE: When I use print(recv_data) without decode function, It prints my unicde data such as :

b'm01,2,130,12_18,\xd9\x88\xd9\x8e\xd9\x85\xd9\x8e\xd9\x86 \xd9\x8a\xd9\x8e\xd8\xb1\xd9\x92\xd8\xba\xd9\x8e\xd8\xa8\xd9\x8f \xd8\xb9\xd9\x8e\xd9\x86 \xd9\x85\xd9\x91\xd9\x90\xd9\x84\xd9\x91\xd9\x8e\xd8\xa9\xd9\x90 \xd8\xa5\xd9\x90\xd8\xa8\xd9\x92\xd8\xb1\xd9\x8e\xd9\xb0\xd9\x87\xd9\x90\xdb\xa6\xd9\x85\xd9\x8e \xd8\xa5\xd9\x90\xd9\x84\xd9\x91\xd9\x8e\xd8\xa7 \xd9\x85\xd9\x8e\xd9\x86 \xd8\xb3\xd9\x8e\xd9\x81\xd9\x90\xd9\x87\xd9\x8e \xd9\x86\xd9\x8e\xd9\x81\xd9\x92\xd8\xb3\xd9\x8e\xd9\x87\xd9\x8f\xdb\xa5 \xd9\x88\xd9\x8e\xd9\x84\xd9\x8e\xd9\x82\xd9\x8e\xd8\xaf\xd9\x90 \xd9\xb1\xd8\xb5\xd9\x92\xd8\xb7\xd9\x8e\xd9\x81\xd9\x8e\xd9\x8a\xd9\x92\xd9\x86\xd9\x8e\xd9\xb0\xd9\x87\xd9\x8f \xd9\x81\xd9\x90\xd9\x89 \xd9\xb1\xd9\x84\xd8\xaf\xd9\x91\xd9\x8f\xd9\x86\xd9\x92\xd9\x8a\xd9\x8e\xd8\xa7 \xd9\x88\xd9\x8e\xd8\xa5\xd9\x90\xd9\x86\xd9\x91\xd9\x8e\xd9\x87\xd9\x8f\xdb\xa5 \xd9\x81\xd9\x90\xd9\x89 \xd9\xb1\xd9\x84\xd9\x92\xd8\xa1\xd9\x8e\xd8\xa7\xd8\xae\xd9\x90\xd8\xb1\xd9\x8e\xd8\xa9\xd9\x90 \xd9\x84\xd9\x8e\xd9\x85\xd9\x90\xd9\x86\xd9\x8e \xd9\xb1\xd9\x84\xd8\xb5\xd9\x91\xd9\x8e\xd9\xb0\xd9\x84\xd9\x90\xd8\xad\xd9\x90\xd9\x8a\xd9\x86\xd9\x8e#m02,2,238,3_4,\xd8\xad\xd9\x8e\xd9\xb0\xd9\x81\xd9\x90\xd8\xb8\xd9\x8f\xd9\x88\xd8\xa7\xdb\x9f \xd8\xb9\xd9\x8e\xd9\x84\xd9\x8e\xd9\x89 \xd9\xb1\xd9\x84\xd8\xb5\xd9\x91\xd9\x8e\xd9\x84\xd9\x8e\xd9\x88\xd9\x8e\xd9\xb0\xd8\xaa\xd9\x90 \xd9\x88\xd9\x8e\xd9\xb1\xd9\x84\xd8\xb5\xd9\x91\xd9\x8e\xd9\x84\xd9\x8e\xd9\x88\xd9\xb0\xd8\xa9\xd9\x90 \xd9\xb1\xd9\x84\xd9\x92\xd9\x88\xd9\x8f\xd8\xb3\xd9\x92\xd8\xb7\xd9\x8e\xd9\x89\xd9\xb0 \xd9\x88\xd9\x8e\xd9\x82\xd9\x8f\xd9\x88\xd9\x85\xd9\x8f\xd9\x88\xd8\xa7\xdb\x9f \xd9\x84\xd9\x90\xd9\x84\xd9\x91\xd9\x8e\xd9\x87\xd9\x90 \xd9\x82\xd9\x8e\xd9\xb0\xd9\x86\xd9\x90\xd8\xaa\xd9\x90\xd9\x8a\xd9\x86\xd9\x8e#m03,2,277,5_7,\xd8\xa5\xd9\x90\xd9\x86\xd9\x91\xd9\x8e \xd9\xb1\xd9\x84\xd9\x91\xd9\x8e\xd8\xb0\xd9\x90\xd9\x8a\xd9\x86\xd9\x8e \xd8\xa1\xd9\x8e\xd8\xa7\xd9\x85\xd9\x8e\xd9\x86\xd9\x8f\xd9\x88\xd8\xa7\xdb\x9f \xd9\x88\xd9\x8e\xd8\xb9\xd9\x8e\xd9\x85\xd9\x90\xd9\x84\xd9\x8f\xd9\x88\xd8\xa7\xdb\x9f \xd9\xb1\xd9\x84\xd8\xb5\xd9\x91\xd9\x8e\xd9\xb0\xd9\x84\xd9\x90\xd8\xad\xd9\x8e\xd9\xb0\xd8\xaa\xd9\x90 \xd9\x88\xd9\x8e\xd8\xa3\xd9\x8e\xd9\x82\xd9\x8e\xd8\xa7\xd9\x85\xd9\x8f\xd9\x88\xd8\xa7\xdb\x9f \xd9\xb1\xd9\x84\xd8\xb5\xd9\x91\xd9\x8e\xd9\x84\xd9\x8e\xd9\x88\xd9\xb0\xd8\xa9\xd9\x8e \xd9\x88\xd9\x8e\xd8\xa1\xd9\x8e\xd8\xa7\xd8\xaa\xd9\x8e\xd9\x88\xd9\x8f\xd8\xa7\xdb\x9f \xd9\xb1\xd9\x84\xd8\xb2\xd9\x91\xd9\x8e\xd9\x83\xd9\x8e\xd9\x88\xd9\xb0\xd8\xa9\xd9\x8e \xd9\x84\xd9\x8e\xd9\x87\xd9\x8f\xd9\x85\xd9\x92 \xd8\xa3\xd9\x8e\xd8\xac\xd9\x92\xd8\xb1\xd9\x8f\xd9\x87\xd9\x8f\xd9\x85\xd9\x92 \xd8\xb9\xd9\x90\xd9\x86\xd8\xaf\xd9\x8e \xd8\xb1\xd9\x8e\xd8\xa8\xd9\x91\xd9\x90\xd9\x87\xd9\x90\xd9\x85\xd9\x92 \xd9\x88\xd9\x8e\xd9\x84\xd9\x8e\xd8\xa7 \xd8\xae\xd9\x8e\xd9\x88\xd9\x92\xd9\x81\xd9\x8c \xd8\xb9\xd9\x8e\xd9\x84\xd9\x8e\xd9\x8a\xd9\x92\xd9\x87\xd9\x90\xd9\x85\xd9\x92 \xd9\x88\xd9\x8e\xd9\x84\xd9\x8e\xd8\xa7 \xd9\x87\xd9\x8f\xd9\x85\xd9\x92 \xd9\x8a\xd9\x8e\xd8\xad\xd9\x92\xd8\xb2\xd9\x8e\xd9\x86\xd9\x8f\xd9\x88\xd9\x86\xd9\x8e#m04,3,39,5_20,\xd9\x81\xd9\x8e\xd9\x86\xd9\x8e\xd8\xa7\xd8\xaf\xd9\x8e\xd8\xaa\xd9\x92\xd9\x87\xd9\x8f \xd9\xb1\xd9\x84\xd9\x92\xd9\x85\xd9\x8e\xd9\x84\xd9\x8e\xd9\xb0\xd9\x93\xd8\xa6\xd9\x90\xd9\x83\xd9\x8e\xd8\xa9\xd9\x8f \xd9\x88\xd9\x8e\xd9\x87\xd9\x8f\xd9\x88\xd9\x8e \xd9\x82\xd9\x8e\xd8\xa7\xd9\x93\xd8\xa6\xd9\x90\xd9\x85\xd9\x8c \xd9\x8a\xd9\x8f\xd8\xb5\xd9\x8e\xd9\x84\xd9\x91\xd9\x90\xd9\x89 \xd9\x81\xd9\x90\xd9\x89 \xd9\xb1\xd9\x84\xd9\x92\xd9\x85\xd9\x90\xd8\xad\xd9\x92\xd8\xb1\xd9\x8e\xd8\xa7\xd8\xa8\xd9\x90 \xd8\xa3\xd9\x8e\xd9\x86\xd9\x91\xd9\x8e \xd9\xb1\xd9\x84\xd9\x84\xd9\x91\xd9\x8e\xd9\x87\xd9\x8e \xd9\x8a\xd9\x8f\xd8\xa8\xd9\x8e\xd8\xb4\xd9\x91\xd9\x90\xd8\xb1\xd9\x8f\xd9\x83\xd9\x8e \xd8\xa8\xd9\x90\xd9\x8a\xd9\x8e\xd8\xad\xd9\x92\xd9\x8a\xd9\x8e\xd9\x89\xd9\xb0 \xd9\x85\xd9\x8f\xd8\xb5\xd9\x8e\xd8\xaf\xd9\x91\xd9\x90\xd9\x82\xd9\x8b\xdb\xa2\xd8\xa7 \xd8\xa8\xd9\x90\xd9\x83\xd9\x8e\xd9\x84\xd9\x90\xd9\x85\xd9\x8e\xd8\xa9\xd9\x8d \xd9\x85\xd9\x91\xd9\x90\xd9\x86\xd9\x8e \xd9\xb1\xd9\x84\xd9\x84\xd9\x91\xd9\x8e\xd9\x87\xd9\x90 \xd9\x88\xd9\x8e\xd8\xb3\xd9\x8e\xd9\x8a\xd9\x91\xd9\x90\xd8\xaf\xd9\x8b\xd8\xa7 \xd9\x88\xd9\x8e\xd8\xad\xd9\x8e\xd8\xb5\xd9\x8f\xd9\x88\xd8\xb1\xd9\x8b\xd8\xa7 \xd9\x88\xd9\x8e\xd9\x86\xd9\x8e\xd8\xa8\xd9\x90\xd9\x8a\xd9\x91\xd9\x8b\xd8\xa7 \xd9\x85\xd9\x91\xd9\x90\xd9\x86\xd9\x8e \xd9\xb1\xd9\x84\xd8\xb5\xd9\x91\xd9\x8e\xd9\xb0\xd9\x84\xd9\x90\xd8\xad\xd9\x90\xd9\x8a\xd9\x86\xd9\x8e#m05,4,77,39_10,\xd8\xa3\xd9\x8

When I comment line recv_data += recv_data Error turn into:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 65535: unexpected end of data

Answer 1

The number 65535 is a big red flag.

Have you noticed that 65535 is 2 to the power of 16 minus one? There are 65536 bytes in 64KiB and I'm betting your socket connection doesn't allow you to send more than 64KiB in one go.

Try sending a smaller set of data and see if you have the same problem. If so, you need to figure out how to send the data set in chunks, put them back together at the receiving end and decode the whole.