Problem sending binary files via sockets, python

Question

I'm trying to write a program that transfers binary files from the client to the server. Here's the code:

Client (send file)

  def send_file(self,filename):
        print("Sending: " + filename)
        size = self.BUFFER_SIZE
        with open(filename,'rb') as f:
            raw = f.read().decode()
        buffer = [raw[i:i + size] for i in range(0, len(raw), size)]
        for x in range(len(buffer)):
            self.sock.sendall(buffer[x].encode())

        return

Server (recv file)

def recv_file(self, conn, filename):
    packet = ""
    buffer = ""
    while True:
        buffer = conn.recv(self.BUFFER_SIZE)
        packet = packet + str(buffer.decode())
        if not len(buffer) == self.BUFFER_SIZE:
            break
    with open(filename, 'wb') as f:
        f.write(bytes(packet.encode()))
    #print(packet)
    return 

This way I can transfer txt files, but when I have to transfer jpeg or any other type of file, it freezes in the loop. Can someone please explain me why? I'm new to py and i'm trying to learn

Answer 1

It shouldn't freeze if both sides have the same locale encoding, but it could easily die with an exception.

You're reading and sending as binary (good), but inexplicably decode -ing to str , then encode ing back to bytes (bad). Problem is, arbitrary binary data isn't guaranteed to be decodable in any given locale; if your locale encoding is UTF-8, odds are it's not legal. If it's latin-1 it's legal, but pointless.

Worse, if your client and server have different locale encodings, the result of decoding might be different on each side (and therefore the lengths won't match).

Use bytes consistently, don't convert to and from strings, and locale settings won't matter. Your code will also run faster. You also need to actually send the file length ahead of time; your loop is hoping recv will return a short length only when the file is done, but if:

1. The file is an exact multiple of the buffer size, or
2. The socket happens to send data in chunks that don't match the buffer size

you can each get short recv results, by coincidence in case #2, and deterministically in case #1.

A safer approach is to actually prefix your transmission with the file length, rather than hoping the chunking works as expected:

def send_file(self,filename):
    print("Sending:", filename)
    with open(filename, 'rb') as f:
        raw = f.read()
    # Send actual length ahead of data, with fixed byteorder and size
    self.sock.sendall(len(raw).to_bytes(8, 'big'))
    # You have the whole thing in memory anyway; don't bother chunking
    self.sock.sendall(raw)

def recv_file(self, conn, filename):
    # Get the expected length (eight bytes long, always)
    expected_size = b""
    while len(expected_size) < 8:
        more_size = conn.recv(8 - len(expected_size))
        if not more_size:
            raise Exception("Short file length received")
        expected_size += more_size

    # Convert to int, the expected file length
    expected_size = int.from_bytes(expected_size, 'big')

    # Until we've received the expected amount of data, keep receiving
    packet = b""  # Use bytes, not str, to accumulate
    while len(packet) < expected_size:
        buffer = conn.recv(expected_size - len(packet))
        if not buffer:
            raise Exception("Incomplete file received")
        packet += buffer
    with open(filename, 'wb') as f:
        f.write(packet)

Answer 2

As an addendum to ShadowRanger's post, If you do want to maintain the file chunking without using socket.sendfile you can utilize a few tricks to clean up your code and reduce memory footprint.

The sending process is fairly simple as we copied the process of sending the file size from ShadowRanger, and added a very simple loop to send chunks of data until the chunk comes up empty (end of the file).

def send_file(self,filename):
    print("Sending: " + filename)
    #send file size as big endian 64 bit value (8 bytes)
    self.sock.sendall(os.stat(filename).st_size.tobytes(8,'big'))
    with open(filename,'rb') as f: #open our file to read
        while True:
            chunk = f.read(self.BUFFER_SIZE) #get next chunk
            if not chunk: #empty chunk indicates EOF
                break
            self.sock.sendall(chunk) #send the chunk

Receiving a file is also very straightforward with the same process to read the expected file size at the beginning, then a loop to read data into that file until we reach our expected size. We then use f.tell() as we receive data as an easy way to tell if the whole file has been sent yet.

def recv_file(self, conn, filename):
    # file size transfer copied from ShadowRanger
    # Get the expected length (eight bytes long, always)
    expected_size = b"" #buffer to read in file size
    while len(expected_size) < 8: #while buffer is smaller than 8 bytes
        more_size = conn.recv(8 - len(expected_size)) #read up to remaining bytes
        if not more_size: #nothing was read
            raise Exception("Short file length received")
        expected_size += more_size #extend buffer
    expected_size = int.from_bytes(expected_size, 'big') #Convert to int, the expected file length
    with open(filename, 'wb') as f: #open our file to write
        while f.tell() < expected_size: #while it's smaller than our expected size
            bytes_recvd = conn.recv() #read any available data 
            f.write(bytes_recvd)