C++ byte array (struct) interpreted by Python

Question

I am trying to pass a C++ struct from my arduino to my raspberry pi. I have a struct that looks like this:

struct node_status
{
  char *node_type = "incubator";
  char *sub_type;          // set the type of incubator
  int sub_type_id;
  bool sleep = false;         // set to sleep
  int check_in_time = 1000;   // set check in time
  bool LOCK = false;      // set if admin control is true/false
} nodeStatus;

I tried using the python module named struct

from struct import *

                print("Rcvd Node Status msg from 0{:o}".format(header.from_node))
                print("node_type: {}".format(unpack("10s",payload[0]))) #node_type
                node_type = unpack("10s",payload[0])
                print("sub_type: {}".format(unpack("10s",payload[1]), header.from_node))    #sub_type
                sub_type = unpack("10s",payload[1])
                print("sub_type_id: {}".format(unpack("b",payload[2])))
                sub_type_id = unpack("b",payload[2])
                print("sleep: {}".format(unpack("?",payload)[3]))   #sleep
                sleep = unpack("?",payload[3])
                print("check_in_time: {}".format(unpack("l",payload[4])))   #check_in_time
                check_in_time = unpack("l",payload[4])
                print("Lock: {}".format(unpack("?",payload[5])))    #LOCK
                Lock = unpack("?",payload[5])

but I am not having much luck. I was even looking at just using ctypes module but seem to not be going anywhere..

from ctypes import *

class interpret_nodes_status(Structure):
    _fields_ = [('node_type',c_char_p),
                ('sub_type',c_char_p),
                ('sub_type_id',c_int),
                ('sleep',c_bool),
                (check_in_time',c_int),
                ('LOCK',c_bool)]

nodestatus = translate_nodes_status(payload)

but that just gives me an error

TypeError: bytes or integer address expected instead of bytearray instance

What can I do? WHERE am I going wrong with this?

EDIT:

I am using the RF24Mesh Library from

https://github.com/nRF24/RF24Mesh

The way I send the message is this?

  RF24NetworkHeader header();

  if (!mesh.write(&nodeStatus, /*type*/ 126, sizeof(nodeStatus), /*to node*/ 000))
  { // Send the data
    if ( !mesh.checkConnection() )
    {
      Serial.println("Renewing Address");
      mesh.renewAddress();

    }
  }
  else
  {
    Serial.println("node status msg Sent");
    return;
  }

}

Answer 1

Your C program is just sending the struct, but the struct doesn't contain any of the string data. It only includes pointers (addresses) which are not usable by any other process (different address spaces).

You would need to determine a way to send all the required data, which would likely mean sending the length of each string and its data.

One way to do that would be to use a maximum length and just store the strings in your struct:

struct node_status
{
  char node_type[48];
  char sub_type[48];          // set the type of incubator
  int sub_type_id;
  bool sleep = false;         // set to sleep
  int check_in_time = 1000;   // set check in time
  bool LOCK = false;      // set if admin control is true/false
} nodeStatus;

You would then need to copy strings into those buffers instead of assigning them, and check for buffer overflow. If the strings are ever entered by users, this has security implications.

Another approach is to pack the data into a single block just when you send it. You could use multiple writes, as well, but I don't know this mesh library or how you would set the type parameter to do that. Using a buffer is something like:

// be sure to check for null on your strings, too.
int lennodetype = strlen(nodeStatus.node_type);
int lensubtype = strlen(nodeStatus.sub_type);
int bufsize = sizeof(nodeStatus) + lennodetype + lensubtype;
byte* buffer = new byte[bufsize];

int offset = 0;
memcpy(buffer+offset, &lennodetype, sizeof(int));
offset += sizeof(int);
memcpy(buffer+offset, nodeStatus.node_type, lennodetype * sizeof(char));
offset += lennodetype * sizeof(char);
memcpy(buffer+offset, &lensubtype, sizeof(int));
offset += sizeof(int);
memcpy(buffer+offset, nodeStatus.sub_type, lensubtype * sizeof(char));
offset += lensubtype * sizeof(char);
// this still copies the pointers, which aren't needed, but simplifies the code
// and 8 unused bytes shouldn't matter too much. You could adjust this line to 
// eliminate it if you wanted.
memcpy(buffer+offset, &nodeStatus, sizeof(nodeStatus));

if (!mesh.write(buffer, 
/*type*/ 126, 
bufsize, 
/*to node*/ 000))
  { // Send the data
    if ( !mesh.checkConnection() )
    {
      Serial.println("Renewing Address");
      mesh.renewAddress();

    }
  }
  else
  {
    Serial.println("node status msg Sent");
  }
delete [] buffer;

Now that the data is actually SENT (a prerequisite for reading the data) the data you need should all be in the payload array. You will need to unpack it, but you can't just pass unpack a single byte, it needs the array:

len = struct.unpack("@4i", payload)
offset = 4
node_type = struct.unpack_from("{}s".format(len), payload, offset)
offset += len
len = struct.unpack_from("@4i", payload, offset)
offset += 4
sub_type = struct.unpack_from("{}s".format(len), payload, offset)
offset += len
...

Answer 2

I might be wrong but I think you don't need to use ctypes, just use the data that you received in python directly -- it might be already packed as a tuple. I wanted to add this as a comment but I don't have enough reputation

Answer 3

I upvoted Garr Godfrey's answer as it is a good one indeed. However, it will increase the struct's size. This neither a good nor bad thing, however if for some reason you would like to keep the solution based on char* pointers instead of arrays (eg you don't know the maximum length of the strings), it can be achieved the following way (my code makes assumption of int's size being 4 bytes, little endian, bool's size=1bytes, char size=1byte):

//_Static_assert(sizeof(int)==4u, "Int size has to be 4 bytes");
//the above one is C11, the one below is C++: 
//feel free to ifdef that if you need it
static_assert(sizeof(int)==4u, "Int size has to be 4 bytes");

struct node_status
{
  char* node_type;
  char* sub_type;          // set the type of incubator
  int sub_type_id;
  bool sleep;         // set to sleep
  int check_in_time;   // set check in time
  bool LOCK;      // set if admin control is true/false
};  


size_t serialize_node_status(const struct node_status* st, char* buffer)
{                                        
    //this bases on the assumption buffer is large enough
    //and string pointers are not null
    size_t offset=0u;                    
    size_t l = 0;                        
                                         
    l = strlen(st->node_type)+1;         
    memcpy(buffer+offset, st->node_type, l); 
    offset += l;                         
                                         
    l = strlen(st->sub_type)+1;          
    memcpy(buffer+offset, st->sub_type, l); 
    offset += l;                         
                                         
    l = sizeof(st->sub_type_id);         
    memcpy(buffer+offset, &st->sub_type_id, l); 
    offset += l;                         
                                         
    l = sizeof(st->sleep);               
    memcpy(buffer+offset, &st->sleep, l);
    offset += l;                         
                                         
    l = sizeof(st->check_in_time);       
    memcpy(buffer+offset, &st->check_in_time, l);                                                                                                                            
    offset += l;                         
                                                                         
    l = sizeof(st->LOCK);                
    memcpy(buffer+offset, &st->LOCK, l); 
    offset += l;                         
                                         
    return offset;    

// sending:
char buf[100] = {0}; //pick the needed size or allocate it dynamically
    
struct node_status nodeStatus = {"abcz", "x", 20, true, 999, false};
size_t serialized_bytes = serialize_node_status(&nodeStatus, buf);

mesh.write(buf, /*type*/ 126, serialized_bytes, /*to node*/ 000);                   

Side note: assigning string literals directly to char pointers is not valid C++. So the string types either should be const char* , eg const char* node_type or the file should be compiled as C (where you can get away with it). Arduino often tends to have its own compilation options set, so it is likely to work due to compiler extension (or just inhibited warning). Thus, not being sure what exactly is going to be used, I wrote a C11-compatible version.

And then on Python's end:

INT_SIZE=4

class node_status:
    def __init__(self,
            nt: str,
            st: str,
            stid: int,
            sl: bool,
            cit: int,
            lck: bool):
        self.node_type = nt
        self.sub_type = st
        self.sub_type_id = stid
        self.sleep = sl
        self.check_in_time  = cit 
        self.LOCK = lck 
 
    def __str__(self):
        s=f'node_type={self.node_type} sub_type={self.sub_type}'
        s+=f' sub_type_id={self.sub_type_id} sleep={self.sleep}'
        s+=f' check_in_time={self.check_in_time} LOCK={self.LOCK}'
        return s;
 
    @classmethod 
    def from_bytes(cls, b: bytes):
        offset = b.index(0x00)+1
        nt = str(b[:offset], 'utf-8')
        b=b[offset:]
        offset = b.index(0x00)+1
        st = str(b[:offset], 'utf-8')
        b=b[offset:]
 
        stid = int.from_bytes(b[:INT_SIZE], 'little')
        b = b[INT_SIZE:]
 
        sl = bool(b[0])
        b = b[1:]
 
        cit = int.from_bytes(b[:INT_SIZE], 'little')
        b = b[INT_SIZE:]
 
        lck = bool(b[0])
        b = b[1:]
 
        assert(len(b) == 0)
 
        return cls(nt, st, stid, sl, cit, lck)             


#and the deserialization goes like this:
    fromMesh1 = bytes([0x61,0x62,0x63,0x0,0x78,0x79,0x7A,0x0,0x14,0x0,0x0,0x0,0x1,0xE7,0x3,0x0,0x0,0x1])
        
    fromMesh2 = bytes([0x61,0x62,0x63,0x0,0x78,0x79,0x7A,0x0,0x14,0x0,0x0,0x0,0x1,0xE7,0x3,0x0,0x0,0x0])
    fromMesh3 = bytes([0x61,0x62,0x63,0x7A,0x0,0x78,0x0,0x14,0x0,0x0,0x0,0x1,0xE7,0x3,0x0,0x0,0x0])
    print(node_status.from_bytes(fromMesh1))
    print(node_status.from_bytes(fromMesh2))
    print(node_status.from_bytes(fromMesh3))                                                                                                                  

Answer 4

These are all good answers but not what was required. I suppose a more in depth knowledge of the RF24Mesh library was needed. I have been able to find the answer with the help of some RF24 pro's. Here is my solution:

I had to change the struct to specific sizes using char name[10] on the C++ arduino side.

struct node_status
{
  char node_type[10] = "incubator";
  char sub_type[10] = "chicken";   // set the type of incubator
  int sub_type_id = 1;
  bool sleep = false;         // set to sleep
  int check_in_time = 1000;   // set check in time
  bool LOCK = false;      // set if admin control is true/false
} nodeStatus;

Unfortunately, it looks like read() returns the payload with a length of what you passed to the read() function. This is unintuitive and should be improved. Not to mention, the parameter specifying the length of the payload to return should be optional.

Until they get a fix for this, I will have to slice the payload to only the length that struct.pack() needs (which can be determined based on the format specifier string). So, basically

# get the max sized payload despite what was actually received
head, payload = network.read(144)

# unpack 30 bytes
(
    node_type,
    sub_type,
    sub_type_id,
    sleep,
    check_in_time,
    LOCK,
) = struct.unpack("<10s10si?i?", payload[:30])

I finally got it to work using this method. I want to be fair about giving the points and would like to have your opinion on who should get them that was closest to this method. Please comment below.