Best way to read data streams with different package sizes from serial port in c++

Question

I am working on firmware of an ATMEL sensor board (accelerometer and gyro)and trying to read the data in a platform in Ubuntu.

Currently the firmware is like this:

Ubuntu sends a character "D" and the firmware in response sends back 20 bytes of data that ends in "\\n" then ubuntu uses serialport_read_until(fd, buff, '\\n') and assumes that buff[0] is byte zero and so on.The frequency of acquisition is 200hz. BUT using this method sometimes I receive corrupted values and it is not working well. Also there are many "Unable to write on serial port" error in ubuntu.

I have found an example code from ATMEL for the firmware and there the data is sent in different packages and continuously (without waiting for the computer to ask for it) the structure is like this:

    void adv_data_send_3(uint8_t stream_num, uint32_t timestamp,
        int32_t value0, int32_t value1, int32_t value2)
{
    /* Define packet format with 3 data fields */
    struct {
        adv_data_start_t start;       /* Starting fields of packet */
        adv_data_field_t field [3];   /* 3 data fields */
        adv_data_end_t end;           /* Ending fields of packet */
    } packet;

    /* Construct packet */
    packet.start.header1 = ADV_PKT_HEADER_1;
    packet.start.header2 = ADV_PKT_HEADER_2;
    packet.start.length  = cpu_to_le16(sizeof(packet));
    packet.start.type    = ADV_PKT_DATA;
    packet.start.stream_num = stream_num;
    packet.start.time_stamp = cpu_to_le32(timestamp);

    packet.field[0].value = cpu_to_le32(value0);
    packet.field[1].value = cpu_to_le32(value1);
    packet.field[2].value = cpu_to_le32(value2);

    packet.end.crc = 0x00;  /* Not used */
    packet.end.mark = ADV_PKT_END;

    /* Write packet */
    adv_write_buf((uint8_t *)&packet, sizeof(packet));
}

but I don't know how I can continuously read the data that is sent in a structure like above.

Sorry if it is a trivial question. I am not a programmer but I need to solve this and I could not find a solution (that I can understand!) after searching for a couple of days.

The reading function I use in linux:

int serialport_read_until(int fd, unsigned char* buf, char until){
char b[1];
int i=0;
do { 
    int n = read(fd, b, 1);  // read a char at a time
    if( n==-1) return -1;    // couldn't read
    if( n==0 ) {
        usleep( 1 * 1000 ); // wait 1 msec try again
        continue;
    }
    buf[i] = b[0]; i++;
} while( b[0] != until );
buf[i] = 0;  // null terminate the string
return 0;}

The new Reading Func:

    // Read the header part
adv_data_start_t start;
serial_read_buf(fd, reinterpret_cast<uint8_t*>(&start), sizeof(start));

// Create a buffer for the data and the end marker
std::vector<uint8_t> data_and_end(start.length - sizeof(start));

// Read the data and end marker
serial_read_buf(fd, data_and_end.data(), data_and_end.size());

// Iterate over the data
size_t num_data_fields = (data_and_end.size() - sizeof(adv_data_end_t)) / sizeof(adv_data_field_t);

adv_data_field_t* fields = reinterpret_cast<adv_data_field_t*>(data_and_end.data());


for (size_t i = 0; i < num_data_fields; i++)
    std::cout << "Field #" << (i + 1) << " = " << fields[i].value << '\n';

The data packets that are sent from the firmware:

typedef struct {
uint8_t         header1;        // header bytes - always 0xFF5A
uint8_t         header2;        // header bytes - always 0xFF5A
uint16_t        length;         // packet length (bytes)
uint32_t        time_stamp;     // time stamp (tick count)
    } adv_data_start_t;


typedef struct {
int32_t         value;          // data field value (3 VALUES)
} adv_data_field_t;


 typedef struct {
uint8_t         crc;            // 8-bit checksum
uint8_t         mark;           // 1-byte end-of-packet marker
uint16_t         mark2;           // 2-byte end-of-packet marker (Added to avoid data structure alignment problem)
 } adv_data_end_t;

Answer 1

Well you have the length of the packet in the packet "header", so read the header fields (the start structure) in one read, and in a second read you read the data and the end.

If the start and end parts are the same for all packets (which I guess they are), you can easily figure out the amount of data fields after the second read.

---

Something like this:

// Read the header part
adv_data_start_t start;
adv_read_buf(reinterpret_cast<uint8_t*>(&start), sizeof(start));

// Create a buffer for the data and the end marker
std::vector<uint8_t> data_and_end(start.length - sizeof(start));

// Read the data and end marker
adv_read_buf(data_and_end.data(), data_and_end.size());

// Iterate over the data
size_t num_data_fields = (data_and_end.size() - sizeof(adv_data_end_t)) / sizeof(adv_data_field_t);

adv_data_end_t* fields = reinterpret_cast<adv_data_end_t*>(data_and_end.data());

for (size_t i = 0; i < num_data_fields; i++)
    std::cout << "Field #" << (i + 1) << " = " << fields[i] << '\n';

---

Possible read_buf implementation:

// Read `bufsize` bytes into `buffer` from a file descriptor
// Will block until `bufsize` bytes has been read
// Returns -1 on error, or `bufsize` on success
int serial_read_buf(int fd, uint8_t* buffer, const size_t bufsize)
{
    uint8_t* current = buffer;
    size_t remaining = bufsize

    while (remaining > 0)
    {
        ssize_t ret = read(fd, current, remaining);

        if (ret == -1)
            return -1;  // Error
        else if (ret == 0)
        {
            // Note: For some descriptors, this means end-of-file or
            //       connection closed.
            usleep(1000);
        }
        else
        {
            current += ret;  // Advance read-point in buffer
            remaining -= ret;  // Less data remaining to read
        }
    }

    return bufsize;
}