Converting / Parsing C++ byte struct to Go

Question

I am reading some data packets in Go, where the fields are C++ data types. I tried parsing the data but I am reading garbage values.

Here is a small example - the data spec sheet for a particular datatype is as follows in C++,

struct CarTelemetryData
{
    uint16    m_speed;                      
    uint8     m_throttle;                   
    int8      m_steer;                      
    uint8     m_brake;                     
    uint8     m_clutch;                     
    int8      m_gear;                       
    uint16    m_engineRPM;                  
    uint8     m_drs;                        
    uint8     m_revLightsPercent;           
    uint16    m_brakesTemperature[4];       
    uint16    m_tyresSurfaceTemperature[4]; 
    uint16    m_tyresInnerTemperature[4];   
    uint16    m_engineTemperature;          
    float     m_tyresPressure[4];           
};

And below is what I have defined in Go

type CarTelemetryData struct {
    Speed                   uint16
    Throttle                uint8
    Steer                   int8
    Brake                   uint8
    Clutch                  uint8
    Gear                    int8
    EngineRPM               uint16
    DRS                     uint8
    RevLightsPercent        uint8
    BrakesTemperature       [4]uint16
    TyresSurfaceTemperature [4]uint16
    TyresInnerTemperature   [4]uint16
    EngineTemperature       uint16
    TyresPressure           [4]float32
}

For the actual un-marshalling, I am doing this -

func decodePayload(dataStruct interface{}, payload []byte) {
    dataReader := bytes.NewReader(payload[:])
    binary.Read(dataReader, binary.LittleEndian, dataStruct)
}

payload := make([]byte, 2048)
s.conn.ReadFromUDP(payload[:])
telemetryData := &data.CarTelemetryData{}
s.PacketsRcvd += 1
decodePayload(telemetryData, payload)

I suspect that this is because the datatypes are not equivalent and there is some conversion issue while reading the bytes into Go data-types, whereas they have been originally packages as C++. How can I deal with this?

Note: I don't have any control over the data that is sent, this is sent by a third party service.

Answer 1

The issue you're facing has to do with the alignment of struct members. You can read more about it here but, in short, the C++ compiler will sometimes add padding bytes in order to maintain the natural alignment expected by the architecture. If that alignment is not used, it may cause degraded performance or even an access violation.

For x86/x64, for example, the alignment of most types will usually (but not necessarily guaranteed to) be the same as the size. We can see that

#include <cstdint>
#include <type_traits>

std::size_t offsets[] = {
    std::alignment_of_v<std::uint8_t>,
    std::alignment_of_v<std::uint16_t>,
    std::alignment_of_v<std::uint32_t>,
    std::alignment_of_v<std::uint64_t>,
    std::alignment_of_v<__uint128_t>,
    std::alignment_of_v<std::int8_t>,
    std::alignment_of_v<std::int16_t>,
    std::alignment_of_v<std::int32_t>,
    std::alignment_of_v<std::int64_t>,
    std::alignment_of_v<__int128_t>,
    std::alignment_of_v<float>,
    std::alignment_of_v<double>,
    std::alignment_of_v<long double>,
    std::alignment_of_v<void*>,
};

compiles to

offsets:
        .quad   1
        .quad   2
        .quad   4
        .quad   8
        .quad   16
        .quad   1
        .quad   2
        .quad   4
        .quad   8
        .quad   16
        .quad   4
        .quad   8
        .quad   16
        .quad   8

Due to these (and other) implementation details, it may be advisable to not rely on the internal representation. In some cases, however, other methods may not be fast enough (such as serializing field by field), or you may not be able to change the C++ code, like OP.

binary.Read expects packed data, but C++ will use padding. We need to either use a compiler-dependent directive such as #pragma pack(1) or add padding the Go struct. The first is not an option for OP, so we'll use the second.

We can use the offsetof macro to determine the offset of a struct member relative to the struct itself. We can do something like

#include <array>
#include <cstddef>
#include <cstdint>

using int8 = std::int8_t;
using uint8 = std::uint8_t;
using uint16 = std::uint16_t;

struct CarTelemetryData {
    uint16 m_speed;
    uint8 m_throttle;
    int8 m_steer;
    uint8 m_brake;
    uint8 m_clutch;
    int8 m_gear;
    uint16 m_engineRPM;
    uint8 m_drs;
    uint8 m_revLightsPercent;
    uint16 m_brakesTemperature[4];
    uint16 m_tyresSurfaceTemperature[4];
    uint16 m_tyresInnerTemperature[4];
    uint16 m_engineTemperature;
    float m_tyresPressure[4];
};

// C++ has no reflection (yet) so we need to list every member
constexpr auto offsets = std::array{
    offsetof(CarTelemetryData, m_speed),
    offsetof(CarTelemetryData, m_throttle),
    offsetof(CarTelemetryData, m_steer),
    offsetof(CarTelemetryData, m_brake),
    offsetof(CarTelemetryData, m_clutch),
    offsetof(CarTelemetryData, m_gear),
    offsetof(CarTelemetryData, m_engineRPM),
    offsetof(CarTelemetryData, m_drs),
    offsetof(CarTelemetryData, m_revLightsPercent),
    offsetof(CarTelemetryData, m_brakesTemperature),
    offsetof(CarTelemetryData, m_tyresSurfaceTemperature),
    offsetof(CarTelemetryData, m_tyresInnerTemperature),
    offsetof(CarTelemetryData, m_engineTemperature),
    offsetof(CarTelemetryData, m_tyresPressure),
};

constexpr auto sizes = std::array{
    sizeof(CarTelemetryData::m_speed),
    sizeof(CarTelemetryData::m_throttle),
    sizeof(CarTelemetryData::m_steer),
    sizeof(CarTelemetryData::m_brake),
    sizeof(CarTelemetryData::m_clutch),
    sizeof(CarTelemetryData::m_gear),
    sizeof(CarTelemetryData::m_engineRPM),
    sizeof(CarTelemetryData::m_drs),
    sizeof(CarTelemetryData::m_revLightsPercent),
    sizeof(CarTelemetryData::m_brakesTemperature),
    sizeof(CarTelemetryData::m_tyresSurfaceTemperature),
    sizeof(CarTelemetryData::m_tyresInnerTemperature),
    sizeof(CarTelemetryData::m_engineTemperature),
    sizeof(CarTelemetryData::m_tyresPressure),
};

constexpr auto computePadding() {
    std::array<std::size_t, offsets.size()> result;

    std::size_t expectedOffset = 0;

    for (std::size_t i = 0; i < offsets.size(); i++) {
        result.at(i) = offsets.at(i) - expectedOffset;
        expectedOffset = offsets.at(i) + sizes.at(i);
    }

    return result;
}

auto padding = computePadding();

which compiles to ( constexpr FTW)

padding:
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   1
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   2

So, on x86, we need one byte before EngineRPM and two bytes before TyresPressure .

So, let's check if that works .

C++:

#include <cstddef>
#include <cstdint>
#include <iomanip>
#include <iostream>
#include <span>

using int8 = std::int8_t;
using uint8 = std::uint8_t;
using uint16 = std::uint16_t;

struct CarTelemetryData {
    uint16 m_speed;
    uint8 m_throttle;
    int8 m_steer;
    uint8 m_brake;
    uint8 m_clutch;
    int8 m_gear;
    uint16 m_engineRPM;
    uint8 m_drs;
    uint8 m_revLightsPercent;
    uint16 m_brakesTemperature[4];
    uint16 m_tyresSurfaceTemperature[4];
    uint16 m_tyresInnerTemperature[4];
    uint16 m_engineTemperature;
    float m_tyresPressure[4];
};

int main() {
    CarTelemetryData data = {
        .m_speed = 1,
        .m_throttle = 2,
        .m_steer = 3,
        .m_brake = 4,
        .m_clutch = 5,
        .m_gear = 6,
        .m_engineRPM = 7,
        .m_drs = 8,
        .m_revLightsPercent = 9,
        .m_brakesTemperature = {10, 11, 12, 13},
        .m_tyresSurfaceTemperature = {14, 15, 16, 17},
        .m_tyresInnerTemperature = {18, 19, 20, 21},
        .m_engineTemperature = 22,
        .m_tyresPressure = {23, 24, 25, 26},
    };

    std::cout << "b := []byte{" << std::hex << std::setfill('0');

    for (auto byte : std::as_bytes(std::span(&data, 1))) {
        std::cout << "0x" << std::setw(2) << static_cast<unsigned>(byte)
                  << ", ";
    }

    std::cout << "}";
}

results in

b := []byte{0x01, 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x00, 0x07, 0x00, 0x08, 0x09, 0x0a, 0x00, 0x0b, 0x00, 0x0c, 0x00, 0x0d, 0x00, 0x0e, 0x00, 0x0f, 0x00, 0x10, 0x00, 0x11, 0x00, 0x12, 0x00, 0x13, 0x00, 0x14, 0x00, 0x15, 0x00, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb8, 0x41, 0x00, 0x00, 0xc0, 0x41, 0x00, 0x00, 0xc8, 0x41, 0x00, 0x00, 0xd0, 0x41, }

Let's use that in Go:

// Type your code here, or load an example.
// Your function name should start with a capital letter.
package main

import (
    "bytes"
    "encoding/binary"
    "fmt"
)

type CarTelemetryData struct {
    Speed                   uint16
    Throttle                uint8
    Steer                   int8
    Brake                   uint8
    Clutch                  uint8
    Gear                    int8
    _                       uint8
    EngineRPM               uint16
    DRS                     uint8
    RevLightsPercent        uint8
    BrakesTemperature       [4]uint16
    TyresSurfaceTemperature [4]uint16
    TyresInnerTemperature   [4]uint16
    EngineTemperature       uint16
    _                       uint16
    TyresPressure           [4]float32
}

func main() {
    b := []byte{0x01, 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x00, 0x07, 0x00, 0x08, 0x09, 0x0a, 0x00, 0x0b, 0x00, 0x0c, 0x00, 0x0d, 0x00, 0x0e, 0x00, 0x0f, 0x00, 0x10, 0x00, 0x11, 0x00, 0x12, 0x00, 0x13, 0x00, 0x14, 0x00, 0x15, 0x00, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb8, 0x41, 0x00, 0x00, 0xc0, 0x41, 0x00, 0x00, 0xc8, 0x41, 0x00, 0x00, 0xd0, 0x41}

    var dataStruct CarTelemetryData

    dataReader := bytes.NewReader(b[:])
    binary.Read(dataReader, binary.LittleEndian, &dataStruct)

    fmt.Printf("%+v", dataStruct)
}

which prints

{Speed:1 Throttle:2 Steer:3 Brake:4 Clutch:5 Gear:6 _:0 EngineRPM:7 DRS:8 RevLightsPercent:9 BrakesTemperature:[10 11 12 13] TyresSurfaceTemperature:[14 15 16 17] TyresInnerTemperature:[18 19 20 21] EngineTemperature:22 _:0 TyresPressure:[23 24 25 26]}

Take the padding bytes out and it fails.