Cross compile mosquitto with existing openssl version on armv7

Question

I've (so far) failed to successfully cross compile mosquitto with TLS for an embedded armv7 device. Without TLS, cross compilation works fine.

Embedded Device Data:

# cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 2 (v7l)
BogoMIPS        : 548.86
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x3
CPU part        : 0xc08
CPU revision    : 2

Hardware        : Generic AM33XX (Flattened Device Tree)
Revision        : 0000
Serial          : 0000000000000000

# openssl version -a
OpenSSL 1.1.1i  8 Dec 2020
not available
platform: linux-armv4
options:  bn(64,32) rc4(char) des(long) idea(int) blowfish(ptr)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG -DOPENSSL_TLS_SECURITY_LEVEL=0
OPENSSLDIR: "/usr/ssl"
ENGINESDIR: "/usr/lib/engines-1.1"
Seeding source: random-device ( "/dev/urandom" "/dev/random" "/dev/hwrng" "/dev/srandom" )

From the embedded device distributor I've gotten the cross compiler, and installed on my machine:

$ mkdir /opt/advantech
$ cd /opt/advantech
$ git clone https://bitbucket.org/bbsmartworx/Toolchains.git
$ sudo dpkg -i Toolchains/deb/*.deb
### The cross compiler is saved under /opt/toolchain/gcc-conel-armv7-linux-gnueabi

Next, download the same version of OpenSSL as found in the embedded device (1.1.1i)

$ wget https://www.openssl.org/source/openssl-1.1.1i.tar.gz
$ tar xfz openssl-1.1.1i.tar.gz
$ cd openssl-1.1.1i
$ ./Configure linux-armv4 --cross-compile-prefix=/opt/toolchain/gcc-conel-armv7-linux-gnueabi/bin/armv7-linux-gnueabi- --prefix=/opt/arm/v7/openssl/ --openssldir=/opt/arm/v7/openssl shared
### I have also tried using the options I got when doing "openssl version -a" on the embedded device,
### but the final result is the same..
### ./Configure linux-armv4 --cross-compile-prefix=/opt/toolchain/gcc-conel-armv7-linux-gnueabi/bin/armv7-linux-gnueabi- --prefix=/opt/arm/v7/openssl/ --openssldir=/opt/arm/v7/openssl shared -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG -DOPENSSL_TLS_SECURITY_LEVEL=0 -fPIC -pthread -Wa,--noexecstack -Wall -O3
$ make
$ make install
$ cd /opt/arm/v7/openssl/bin/
$ file openssl
openssl: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.3, for GNU/Linux 3.12.10, stripped
$ ls -la openssl
-rwxr-xr-x 1 eh eh 554356 Feb 19 09:36 openssl
### The file size on my machine does NOT matches the one on the embedded device:
$ ssh root@embedded-device
# ls -la /usr/bin/openssl
-rwxr-xr-x    1 root     root        535796 Dec 17 10:36 /usr/bin/openssl
# exit
Connection to embedded-device closed.

Move OpenSSLs includes and libraries within the sysroot of cross compiler:

$ cd /opt/toolchain/gcc-conel-armv7-linux-gnueabi/sysroot/usr/lib
$ sudo cp -P /opt/arm/v7/openssl/lib/libcrypto* .
$ sudo cp -P /opt/arm/v7/openssl/lib/libssl* .
$ cd /opt/toolchain/gcc-conel-armv7-linux-gnueabi/sysroot/usr/include
$ sudo cp -r /opt/arm/v7/openssl/include/openssl .

Download mosquitto and cross compile it

$ wget https://mosquitto.org/files/source/mosquitto-2.0.7.tar.gz
$ tar xvfz mosquitto-2.0.7.tar.gz
$ cd mosquitto-2.0.7
$ make CC=/opt/toolchain/gcc-conel-armv7-linux-gnueabi/bin/armv7-linux-gnueabi-gcc CXX=/opt/toolchain/gcc-conel-armv7-linux-gnueabi/bin/armv7-linux-gnueabi-g++ AR=/opt/toolchain/gcc-conel-armv7-linux-gnueabi/bin/armv7-linux-gnueabi-ar LD=/opt/toolchain/gcc-conel-armv7-linux-gnueabi/bin/armv7-linux-gnueabi-ld
$ file src/mosquitto
src/mosquitto: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.3, for GNU/Linux 3.12.10, stripped
$ file lib/libmosquitto.so.1
lib/libmosquitto.so.1: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, stripped

Transfer cross compiled mosquitto and lib to embedded device:

$ scp src/mosquitto root@embedded-device:/opt/mosquitto/bin/.
$ scp lib/libmosquitto.so.1 root@embedded-device:/usr/lib/.
$ ssh root@embedded-device
# cd /opt/mosquitto/bin/
# ./mosquitto
./mosquitto: symbol lookup error: ./mosquitto: undefined symbol: ENGINE_load_builtin_engines, version OPENSSL_1_1_0

As mentioned before, when I modify mosquitto's config.mk to not use TLS and TLS_PSK, it works correctly on the embedded device.

config.mk:

WITH_TLS:=yes        # when set to "no", no errors on embedded device
WITH_TLS_PSK:=yes    # when set to "no", no errors on embedded device
WITH_CJSON:=no       # is always set to no

How should I cross compile OpenSSL to 100% match the version installed on the embedded device?

Changing OpenSSL on embedded device is not an option.

Answer 1

Your target may not have dedicated hardware for encryption, and you therefore may not have any openssl crypto-engine implementation available on your platform, nor a version of openssl compiled with support for crypto-engines.

You can check by executing ls: /usr/lib/engines-1.1 , since this is the location specified in the output for the openssl version -a command you executed. If no dynamic libraries are present, or the directory does not exist, this is likely that you don't have any support for crypto-engines currently available on your system.

In this case, you will have to re-build mosquitto with the CFLAGS=-DOPENSSL_NO_ENGINE option, so that mosquitto will not attempt to load any (non-existing) openssl crypto-engine at startup.