"certificate verify failed" when bridging AWS IoT with local Mosquitto MQTT

Question

I have a mosquitto MQTT on a local raspberry pi working like a charm. I created a MQTT broker on AWS IoT that works as well.

On my raspberry pi I can connect, publish and subscribe on the AWS broker "manually", using the commands mosquitto_pub and mosquitto_sub. When I do this manually, i use all the certificates and stuff. The command I use is:

mosquitto_pub --cafile amazonCA1.pem --cert certificate.cert --key private.key -h XXXXXXXXXXXXXXXXXX.amazonaws.com -p 8883 -q 1 -d -t "iot/test" -m "testing message"

So, I think the problem is not on the certificates.

The problem is when I change the configuration to use "bridge mode" i get the following message on mosquitto log:

1584371971: Connecting bridge (step 1) awsiot (XXXXXXXXXXXXXXXXXXXXX.amazonaws.com:8883)
1584371972: Connecting bridge (step 2) awsiot (XXXXXXXXXXXXXXXXXXXXX.amazonaws.com:8883)
1584371972: Bridge bridgeawsiot sending CONNECT
1584371972: OpenSSL Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
1584371972: Socket error on client local.bridgeawsiot, disconnecting.
1584371977: Bridge local.bridgeawsiot doing local SUBSCRIBE on topic #

Here is my mosquitto.conf:

pid_file /var/run/mosquitto.pid

persistence true persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log log_type all
#log_dest topic

log_type error log_type warning log_type notice log_type information

connection_messages true log_timestamp true

include_dir /etc/mosquitto/conf.d

password_file /etc/mosquitto/passwordfile allow_anonymous false

And here is my /etc/mosquitto/conf.d/bridge.conf

connection awsiot
address XXXXXXXXXXXXXXXXXXXX.amazonaws.com:8883

# Specifying which topics are bridged
topic # both 1

# Setting protocol version explicitly
bridge_protocol_version mqttv311
bridge_insecure false

# Bridge connection name and MQTT client Id,
# enabling the connection automatically when the broker starts.
cleansession true
clientid bridgeawsiot

start_type automatic
notifications false
log_type all


# =================================================================
# Certificate based SSL/TLS support
# -----------------------------------------------------------------
#Path to the rootCA
bridge_cafile /home/pi/certs/amazonCA1.pem

# Path to the PEM encoded client certificate
bridge_certfile /home/pi/certs/certificate.cert

# Path to the PEM encoded client private key
bridge_keyfile /home/pi/certs/private.key

So, overall the problem is: when I connect/publish/subscribe manually, everything works... but when I use the bridge conf file I get the error:

OpenSSL Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

Any suggestions? Is there any problem using authentication method with username/pw on my local broker (raspberry pi) and certificate authentication on AWS??

Thanks

Answer 1

Ok, I have no idea of what I did, I just know it fixed the problem.

At first I did a clean install of mosquitto on a Ubuntu VM I have and everything worked correctly.

Then I uninstalled Mosquitto from my raspberry pi and installed it again. Configured it just the way I configured the Ubuntu VM and still no luck. I started to think the problem was my raspbian image... but after fiddling a little bit on the configurations, moving the certificates files from one directory to another, changing their permissions, changing the bridge.conf file directory and stuff... It started working and now it's ok.

So if youre having this problem in the future: maybe is just the permissions of the files or directories.

EDIT (one day later) : as I tried to replicate the same thing on another broker, I did everything the same but as soon as my local broker stablished the connection with AWS IoT bridge the connection was lost (message below. No certificate error this time):

1584456917: Bridge local.bridgeawsiot doing local SUBSCRIBE on topic #
1584456917: Connecting bridge (step 1) awsiot (XXXXXXXXXXXXXXX.amazonaws.com:8883)
1584456918: Connecting bridge (step 2) awsiot (XXXXXXXXXXXXXXX.amazonaws.com:8883)
1584456918: Bridge bridgeawsiot sending CONNECT
1584456918: Received CONNACK on connection local.bridgeawsiot.
1584456918: Bridge local.bridgeawsiot sending SUBSCRIBE (Mid: 2, Topic: #, QoS: 0, Options: 0x00)
1584456918: Sending PUBLISH to local.bridgeawsiot (d0, q0, r1, m0, 'XXXXX/XXXXX/XXXXX', ... (6 bytes))
1584456918: Sending PUBLISH to local.bridgeawsiot (d0, q0, r1, m0, 'XXXXX/XXXXX/XXXXX/LWT', ... (6 bytes))
1584456918: Sending PUBLISH to local.bridgeawsiot (d0, q0, r1, m0, 'XXXXX/XXXXX/XXXXX/LWT', ... (6 bytes))
1584456918: Sending PUBLISH to local.bridgeawsiot (d0, q0, r1, m0, 'XXXXX/XXXXX/XXXXX/LWT', ... (6 bytes))
1584456918: Sending PUBLISH to local.bridgeawsiot (d0, q0, r1, m0, 'XXXXX/XXXXX/XXXXX/LWT', ... (6 bytes))
1584456918: Sending PUBLISH to local.bridgeawsiot (d0, q0, r1, m0, 'XXXXX/XXXXX/XXXXX/LWT', ... (6 bytes))

1584456918: Received SUBACK from local.bridgeawsiot
1584456919: Socket error on client local.bridgeawsiot, disconnecting.

I was using the bridge with all the topics:

topic # both 1

I THINK as soon as I connected with the bridge many devices published a lot of messages and the connection was dropped. So after I changed the bridged topic everything was correct

topic iot/test both 1

[Another EDIT: 3 days later] I find out why it was disconnecting when I used "topic # both 1": because one of my devices was sending a message with RETAIN flag set to TRUE.

The documentation of AWS IoT says that it doesn't support RETAIN TRUE and if any message are sent that way AWS IoT Broker disconnects.

Answer 2

The rootCA.pem was invalid . Following how-to-bridge-mosquitto-mqtt-broker-to-aws-iot , they reference AmazonRootCA1.pem for the rootCA.pem file. However, that gives an error using openssl for verification:

openssl s_client -connect <endpoint>.iot.us-east-1.amazonaws.com:8443 -CAfile rootCA.pem  -cert cert.crt -key private.key
...
verify error:num=20:unable to get local issuer certificate

There are some clues about that openssl error at OpenSSL Verify return code: 20 (unable to get local issuer certificate) , where there is emphasis about the CAfile path.

In another tutorial to configure the bridge: Arduino-AWS-IOT-Bridge , there is a different reference for the rootCA.pem file: Public-Primary-Certification-Authority-G5.pem . At last, trying the openssl s_client command using that new rootCA.pem returns:

verify return:1