Receive big list through TCP Sockets - Python

Question

Which would be the best way to receive a big list through TCP Sockets ?

My code looks like this. When u have to receive a big list, that doesn't work obviously.

    print 'connection from', client_address
    while True:
        try:
            data = pickle.loads(connection.recv(8192))
        except EOFError:
            print 'no more data from', client_address
            break  

Answer 1

The best way to do this is to transform the socket into a file object. You can do this with connection.makefile() . Then, instead of calling pickle.loads() -- which expects a complete byte string containing the entire pickled object -- call pickle.load(connection.makefile()) . This way, you let the pickle module handle reading the entire "file". It will call the object's read function repeatedly until it has received all the expected data.

That basically assumes that the entire remaining part of the stream is to be unpickled (which appears to be what you're trying to do so it sounds like it should work). Otherwise, you may need to wrap your own pseudo-file-object around the stream that has some independent knowledge of the end of the pickled object.