How to read raw bytes from socket in Python?

Question

I have an android java app sending bytes over a socket which is connected to a host machine running a server in Python. I need to receive these bytes as they were sent from the python socket. I see that in Python 'socket.recv' only returns a string. When I send an ASCII string from the java app, I am able to receive the data correctly in the python server, but when I send binary data using java byte, I see the data received is not same. I need to receive raw bytes in Python for my protocol to work correctly. Please point me in right direction.

Code snippet for Sending data on socket:

private void sendFrameMessage(byte[] data) {
        byte[] lengthInfo = new byte[4];
        Log.v(TAG, "sendFrameMessage");

        for(int i=0; i<data.length; i++) {
            Log.v(TAG, String.format("data[%d] = %d", i, data[i]));
        }

        try {
            lengthInfo[0] = (byte) data.length;
            lengthInfo[1] = (byte) (data.length >> 8);
            lengthInfo[2] = (byte) (data.length >> 16);
            lengthInfo[3] = (byte) (data.length >> 24);
            DataOutputStream dos;
            dos = new DataOutputStream(mSocket.getOutputStream());
            dos.write(lengthInfo, 0, 4);
            dos.write(data, 0, data.length);
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

Python Code on receiver side

def recvFrameMessage(self, s):
        recv_count = 4;
        data = s.recv(recv_count)
        if data == 0:
            return None
        total_rx = len(data)
        lenInfo = data
        while total_rx < recv_count:
            data = s.recv(recv_count - total_rx)
            if data == 0:
                return None
            total_rx += len(data)
            lenInfo = lenInfo + data

        recv_count = self.decodeFrameLen(lenInfo)
        logger.info("length = %d" % recv_count)

        data = s.recv(recv_count)
        total_rx = len(data)
        msg = data
        while total_rx < recv_count:
            data = s.recv(recv_count - total_rx)
            if data == 0:
                return None            
            total_rx += len(data)
            msg = msg + data
        logger.info("msg = " + msg)
        for i in range(0, len(msg)-1):
            logger.info("msg[%d] = %s" % (i, msg[i]))
        return msg

Answer 1

Reading binary data is perfectly doable, but what if the binary representation from your android app is different than the byte representation on the Python server? From the Python documentation:

It is perfectly possible to send binary data over a socket. The major problem is that not all machines use the same formats for binary data. For example, a Motorola chip will represent a 16 bit integer with the value 1 as the two hex bytes 00 01. Intel and DEC, however, are byte-reversed - that same 1 is 01 00. Socket libraries have calls for converting 16 and 32 bit integers - ntohl, htonl, ntohs, htons where “n” means network and “h” means host, “s” means short and “l” means long. Where network order is host order, these do nothing, but where the machine is byte-reversed, these swap the bytes around appropriately.

Without code and example input/output, this question is going to be really difficult to answer. I assume the issue is that the representation is different. The most likely issue is that Java uses big endian, whereas Python adheres to whatever machine you are running it off of. If your server uses little endian, then you need to account for that. See here for a more thorough explanation on endianness .

Answer 2

@SteveP makes good points for binary data "with some structure", but if this is a plain stream of bytes, in Python 2 simply apply the ord() function to each "character" you get from the socket. For example, if the Java end sends a NUL byte, that will show up on the Python end as the character "\\x00" , and then:

>>> ord("\x00")
0

To convert a whole string s ,

map(ord, s)

returns a list of the corresponding 8-bit unsigned integers.

I'm assuming Python 2 here.