将time.sleep添加到多线程程序可以解决python中的UnicodeDecodeError问题

Question

Here's a basic idea of the threads that I am creating in my program:

 Main thread
        |
   ListenerCreator(The WebSocketServer thread)  ---> Several listener threads(using log())

So the main thread creates a ListenerCreator thread, which connects to a number of clients and creates a listener thread for each client. Here's briefly what a listener thread does: EDIT1 : I'm using WebSockets to read/write data off my client. I've made my own server for this purpose. There is a framing protocol which the standard specifies -- and I am using that. On the client side I am simply using WebSocket.send() and "unmasking" the messages according to the instructions given in the protocol(see section 5.3 in the link above). I would be willing to provide the server code if someone requests it, however, here's a brief outline:

class WebSocketServer:
    def start(): 
          #Open server socket, bind to host:port
          while True:
              #Accept client socket, start a new listener thread for self.log(client)
    def log(client):
          #Receive data using socket.socket.recv(1024)
          #Unmask data as per the protocol
          #Decode using data.decode("utf-8")
          #Append to data_q while holding data_q_lock

There are other methods - those to facilitate sending, closing, handshaking and so on.

Meanwhile in the main thread:

   while breaking!=len(client_list):
        #time.sleep(0.5)    
        with data_q_lock:
           for i in range(len(data_q)):
                mes = data_q.pop()
                for m in client_list:
                    if "#DONE"== mes:
                        breaking += 1
                if(mes[:len("#COUNT:")] == "#COUNT:"):
                    print(mes)

So basically what this loop does is: Loop thru the data_q, if the message starts with "#COUNT", print the message, and after getting a certain number of "#DONE" messages, exit the loop. If the time.sleep is uncommented, then this code works, however without time.sleep I get an UnicodeDecodeError in the log function. Also I only get the error sometimes , sometimes the program works perfectly. (The client is sending the same data every time, by the way) So, my question is, why is the time.sleep required? I thought it was something to do with the GIL in python, as time.sleep releases the GIL. However, even after reading about it I couldn't solve the question

Answer 1

Currently there is no information about how the listener is reading data off the socket. It seems likely however that this is being caused by the usual misunderstanding of sockets.

Data sent down a socket is not "framed" in any way by the socket. Imagine if I sent the message "hello" three times down a socket. Then, like writing to a file without line breaks, the following would flow on the socket:

hellohellohello

Now consider the reader ... when reading the data, how does it know where one message ("hello") starts and and the next? It cannot, unless the sender and receiver agree about how that data should be "framed". This could be done by agreeing on some protocol like:

• null-terminating data; or
• fixed size messages; or
• size prefixed messages.

It gets more complicated of course, even once you've decided how the data should be framed, you cannot guarantee that socket.recv will return a "whole" message ... it will simply return whatever data happens to be in the buffer at the time. It may be a half a message, or a message and a half. Its your job to collate the data read from the socket and divide it into messages.

Turning to your problem, where you are sending utf-8 data. How does the reader know it has read a full utf-8 data message? Most likely, what is happening here is that you have only received a partial message ... there is still more to arrive.

In particular, a valid utf-8 character may consist of more than one byte. So if your partial message ends in the middle of a multi-byte utf-8 representation of a character, then you can certainly not decode it.