无法设置 MQTT (Mosquito Docker) + SSL + MQTTNet (C#) + Dapr.io 绑定

Question

I am trying to set up a local MQTT broker using mosquito image and connect to it with MQTTNet as publisher and subscribe using Dapr MQTT binding. Everything works fine if I am using anonymous mode (both in Docker self-hosted and in K8S). But we need to use SSL... and here things stopped working. And I am literally out of ideas.

Anyone done this setup already?

I am following this guide for creating certificates: Mosquito SSL setup

This is my broker Dockerfile:

FROM eclipse-mosquitto:latest

COPY ./mqtt/mosquitto/mosquitto.conf /mosquitto/config/mosquitto.conf
COPY ./mqtt/mosquitto/certificates /mosquitto/certs
COPY ./mqtt/mosquitto/certificates /mosquitto/ca_certificates

and mosquitto.conf file:

listener 8883
protocol mqtt

listener 9001
protocol websockets

allow_anonymous false

cafile ./mosquitto/certs/ca.crt
keyfile ./mosquitto/certs/server.key
certfile ./mosquitto/certs/server.crt

require_certificate true
use_identity_as_username true

tls_version tlsv1.2

Broker is starting fine:

2023-01-02 23:44:16 1672699456: mosquitto version 2.0.15 starting
2023-01-02 23:44:16 1672699456: Config loaded from /mosquitto/config/mosquitto.conf.
2023-01-02 23:44:16 1672699456: Opening ipv4 listen socket on port 8883.
2023-01-02 23:44:16 1672699456: Opening ipv6 listen socket on port 8883.
2023-01-02 23:44:16 1672699456: Opening websockets listen socket on port 9001.
2023-01-02 23:44:16 1672699456: mosquitto version 2.0.15 running

Now, I am trying to use MQTTNet library and I wrote a simple c# console app:

using MQTTnet;
using MQTTnet.Client;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Text.Json;

var caCert = new X509Certificate2("ca.crt");
var clientCert = new X509Certificate2("client.pfx", "password");

var url = "localhost";
var factory = new MqttFactory();
var client = factory.CreateMqttClient();
var options = new MqttClientOptionsBuilder()
    .WithTcpServer(url, 8883)
    .WithClientId("Device1")
    .WithTls(new MqttClientOptionsBuilderTlsParameters
    {
        UseTls = true,
        SslProtocol = System.Security.Authentication.SslProtocols.Tls12,
        CertificateValidationHandler = (certContext) =>
        {
            X509Chain chain = new X509Chain();
            chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
            chain.ChainPolicy.RevocationFlag = X509RevocationFlag.ExcludeRoot;
            chain.ChainPolicy.VerificationFlags = X509VerificationFlags.NoFlag;
            chain.ChainPolicy.VerificationTime = DateTime.Now;
            chain.ChainPolicy.UrlRetrievalTimeout = new TimeSpan(0, 0, 0);
            chain.ChainPolicy.CustomTrustStore.Add(caCert);
            chain.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;

            var x5092 = new X509Certificate2(certContext.Certificate);

            return chain.Build(x5092);
        },
        AllowUntrustedCertificates = true,
        Certificates = new List<X509Certificate2>
        {
            clientCert, caCert
        },
    })
    .Build();

await client.ConnectAsync(options);

var message = new MqttApplicationMessageBuilder()
    .WithTopic("sensor/reporting")
    .WithPayload(Encoding.UTF8.GetBytes("test"))
    .Build();
await client.PublishAsync(message);

And I am not able to connect with client error:

MQT.net.Exceptions.MqttCommunicationException: ' Received an unexpected EOF or 0 bytes from the transport stream.'

And server error:

2023-01-02 23:47:00 1672699620: New connection from 172.19.0.1:37912 on port 8883.
2023-01-02 23:47:00 1672699620: Client disconnected due to protocol error.

Tried different options and never succeded.

On the subscriber side, I am following this guide: Dapr MQTT Binding with below config file:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: sensorreporting
spec:
  type: bindings.mqtt
  version: v1
  metadata:
  - name: url
    value: ssl://localhost:8883
  - name: topic
    value: sensor/reporting
  - name: consumerID
    value: "{uuid}"
  - name: caCert
    value: -----BEGIN CERTIFICATE-----\nMIIEGzCCAwOgAwIBAgIUYbZRJpxF3knVBs4A9FIc9KU6YskwDQYJKoZIhvcNAQEL\nBQAwgZwxCzAJBgNVBAYTAkNBMRAwDgYDVQQIDAdBbGJlcnRhMRAwDgYDVQQHDAdD\nYWxnYXJ5MRMwEQYDVQQKDAptQ2xvdWRDb3JwMRYwFAYDVQQLDA1DZXJ0QXV0aEFk\nbWluMQ0wCwYDVQQDDARhZGNhMS0wKwYJKoZIhvcNAQkBFh5taWNoYWwua29zb3dz\na2lAbWNsb3VkY29ycC5jb20wHhcNMjMwMTAyMjE0MzU5WhcNMjgwMTAyMjE0MzU5\nWjCBnDELMAkGA1UEBhMCQ0ExEDAOBgNVBAgMB0FsYmVydGExEDAOBgNVBAcMB0Nh\nbGdhcnkxEzARBgNVBAoMCm1DbG91ZENvcnAxFjAUBgNVBAsMDUNlcnRBdXRoQWRt\naW4xDTALBgNVBAMMBGFkY2ExLTArBgkqhkiG9w0BCQEWHm1pY2hhbC5rb3Nvd3Nr\naUBtY2xvdWRjb3JwLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB\nAMxZASmaTBj+8nVnQwngdjQ3632AVforWdZWSW7a/Z9mhlCis8Lmg/B7Hun9NUh3\nlwg3GxB9RN0K3A+aeHObWfSUl+R1NiTVAMWh+CtUVibjWnDJGk+TNQ9Drq5Lh/iX\nNPc2ztUm+iaFyWdXcthWjeYqsnLD4NHbD0470F4mNidg93cPvB66c0Eam01pAAkR\nQ/jUU0W8gncN3SEHc/FAUahGp1xZzxWhawBAr/oa7xjDMZsz4cLBHjnUH/wNuTrZ\nxQ7g/ArO9DsDaITj7+tzKvOLkCza3LzjW7Ye19XL5l9DisX3xBNCeERIIdndwkOm\ndkEif6QIMgZjaYyyMH6JSsECAwEAAaNTMFEwHQYDVR0OBBYEFDDUsLcvER/gEJgl\nstJ3VutBQ/t+MB8GA1UdIwQYMBaAFDDUsLcvER/gEJglstJ3VutBQ/t+MA8GA1Ud\nEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBALzcZ32VWo37Da5wfFmONo1I\n+bnnP4dc6i90/6MVuLoco6nt63XE+DwL53Ib8tPYVMssKCLOBCI+y+tRtpIAIF+y\ntg41SP5iLIU4eV4rjzdd3iirZOavvcx2cLVfMRHFVr1/8V5o4LaQOpiwVycndcHY\noRTXW6h49YpZt914oEwezSRBrN5h8Rc4cJB/alVpj/2FMq0+C6qQmgJ56xLT9Yu+\nAFz2X4zoFX69WsGm+h/cOfrMjqdR96UoS8cUgEjeNPZCxrZLWGEvphnjmjCom+SQ\nJ+qvQQBCGlNW6Hajd6yBybynB4ImwsqySlWYuX/JwfCDxq1rt++lWjha7hzGSdY=\n-----END CERTIFICATE-----
  - name: clientCert
    value: -----BEGIN CERTIFICATE-----\nMIIDwDCCAqgCFA7Zd6l6GgZYMtdtTwKHM7b3jn0HMA0GCSqGSIb3DQEBCwUAMIGc\nMQswCQYDVQQGEwJDQTEQMA4GA1UECAwHQWxiZXJ0YTEQMA4GA1UEBwwHQ2FsZ2Fy\neTETMBEGA1UECgwKbUNsb3VkQ29ycDEWMBQGA1UECwwNQ2VydEF1dGhBZG1pbjEN\nMAsGA1UEAwwEYWRjYTEtMCsGCSqGSIb3DQEJARYebWljaGFsLmtvc293c2tpQG1j\nbG91ZGNvcnAuY29tMB4XDTIzMDEwMjIxNTcwMVoXDTIzMTIyODIxNTcwMVowgZsx\nCzAJBgNVBAYTAkNBMRAwDgYDVQQIDAdBbGJlcnRhMRAwDgYDVQQHDAdDYWxnYXJ5\nMRMwEQYDVQQKDAptQ2xvdWRDb3JwMRIwEAYDVQQLDAlhZC1jbGllbnQxEDAOBgNV\nBAMMB2RldmljZTExLTArBgkqhkiG9w0BCQEWHm1pY2hhbC5rb3Nvd3NraUBtY2xv\ndWRjb3JwLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOZ2d3jm\n240jQ9JdTAoAEe/LaoFD+0q7TisXLeHBruCZWjMCbiqinT+GtvyhOvNVVJk8/0jH\n1pTkAIads1hIqve6AUNinrZd9LRbW8CNeESz+w29GcOcdZ9fMsQf42PqHd+Y8Aes\nF/2TSU9Qu+dVpplrdHOfz5WjmC88/AD9btMDrrJQOhi4MFD8Buy4S5Lrw4ZRQ4be\n72hHAuD3nVT+/WS9EJtVSANHaIHsOPDmyJAPFVS6wWZRAHv5BEf5D/UtjmGg3VRn\nGN2krK1/MLMSv20kjePi7dErOtAE1Q2fsvEfs0zko3/qrGZlELVlxKqYgnAMfY2w\nsVf8sGvTz9sJIKkCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAc/+wE8Efgmi1GB5G\nSlPpFbDmM+pAXm7cAbELTkQK4i70TZ4R7aRS9QulxSxL1MwpHkyQqvW9bN0s+WtX\nUqh3ERyXCyeuOc6KpskROryyCD03rL2j83KIeTvbLSwsx3gwzE0nBy336Y2A1fYT\nJVkYUhi4Gh9LqSovw8yn+O/DjGhzfBPo1MaPZihtVGmTlhvY9ypUsefggC+FE8E8\n8UmvesY7H+/h5TammDoHTFBupQG9JIx76DZJFQlyyGGanttfBooya4ZLzpZ+zmeW\nFOvz+RmekQzjt+Jrtrzv1zkyrhFB129kr9/cNzZBtR2Hte2ezSgmPHFjIPZ0Z4ZC\nTW47cw==-----END CERTIFICATE-----
  - name: clientKey
    value: -----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA5nZ3eObbjSND0l1MCgAR78tqgUP7SrtOKxct4cGu4JlaMwJu\nKqKdP4a2/KE681VUmTz/SMfWlOQAhp2zWEiq97oBQ2Ketl30tFtbwI14RLP7Db0Z\nw5x1n18yxB/jY+od35jwB6wX/ZNJT1C751WmmWt0c5/PlaOYLzz8AP1u0wOuslA6\nGLgwUPwG7LhLkuvDhlFDht7vaEcC4PedVP79ZL0Qm1VIA0dogew48ObIkA8VVLrB\nZlEAe/kER/kP9S2OYaDdVGcY3aSsrX8wsxK/bSSN4+Lt0Ss60ATVDZ+y8R+zTOSj\nf+qsZmUQtWXEqpiCcAx9jbCxV/ywa9PP2wkgqQIDAQABAoIBAQDP28dztxwMFuNj\nx5v+ZQPIeHot7oemZnthJ/3M4Sh+EDInUajVMWeMVU+TWUPXFn4/26EQHpVuIppK\nz89i79+roQrkoP0u1F0RkliVucLgjEgBoy2qnOIFhRI/i9dGVXDuK31KHjBhF8eG\nDvX69uhU9XLJmjja6Psol+NebTXYgIISUz0S39KGqH3BZ9IcRre/peNDh4LqUot9\n+Wmrj/4kIgwC2FXLWw5Bz5wH4cvZbrSrf+p4jf8MZehBBTlkJ2270aEwU2Di+rDx\nS4o1rzDEB3FO6eoDJk+vxseZhqUjAX7Vm2HDZhEM+y1rXuOXY11ya2phIA7Y4YzX\nBuXvNQKBAoGBAP8PI4wk8XjVt5mD6KUbvPzcmEtTG43PrTKh9vXdsP9bkW8+RUEp\npwpguSxPYCXRGSl4A3Ava/8uT8grAx1978LY7l65uCK9EUTdYs5a3BrjhxBVE3Bj\np/e3VSmE8qD1ixlnfcRUf0Dv8CrivKmW4Lz226A3hS6AlWfzI9amEfG5AoGBAOdQ\nGcKSUijcNkASud4VZyOqNqjKtj5SKx1HblnOq7HERblPW53wD8Mg9jeQ8X7Flaox\n52m1W3EcO1VVLveQ6FYsBXVz+9FwHblztizVlty48XmzpQDbAL1xI+G0WT7cpDt8\nDdnlOmZX4hxBsmrkOFz/gP4yn6sL0u0wbRTk995xAoGAcBe1Jaguv04djEgefEF+\ngxpni+MQXviAQ5pOsKsnWHJjriINyZ+Gm9b6SnDv7m7AwirOSAVvTwyJX40Xp2Rr\ndJRl3hd8xzeUWVF6VPo1dVO9Tf41ttT1Qra9pKua/gYrhZSn/xqrelx4jkBrbZSz\nXBiZWQZNb6xu/OosO+9sP3ECgYEA3KoVaQCdUE2Dk2xJ1Asm8WcX7b40TQ+mAbB7\nHUGDmXpb6YRDKlAxOTzgpfGn8AAd3tbspHNWYHlGQQmaXKHogwxLyNh0ikAimyA/\n0lWQLdBgzHEMH9zMv+yHBnI+ETH7mqOlsxByvd9o3PvvcPl2q4EzVUyF9W66MXvZ\nE+26aIECgYBtDK+8hffO3N1yXW+tMFuCbbzMPrPJC5o4VII8pm+oNcsvW1Ao9kpZ\nL7yVVJCozo0omvuY2ohVchiQ5jC1nxd17Xa0DuIrKgtcEmpoae16e2h7kcSjv63y\nbKEEUIKRyWfhx6iJb8o03lq1UUxRDhaB5leiUUddibJyhh7EDfCHfQ==\n-----END RSA PRIVATE KEY-----

and dapr sidecar is failing with following error:

mqtt-sample-mqtt-api-dapr-1 | time="2023-01-02T23:19:05.428410597Z" level=fatal msg="process component sensorreporting error: [INIT_COMPONENT_FAILURE]: initialization error occurred for sensorreporting (bindings.mqtt/v1): mqtt binding error: invalid ca certificate" app_id=mqtt-api instance=2c64804e20f1 scope=dapr.runtime type=log ver=1.9.5 mqtt-sample-mqtt-api-dapr-1 exited with code 1

I do not know where I am making mistake. I am going to test that on AKS now (maybe the problem is with running it locally? Anonymous mode is working with AKS).

I was able to set up the whole thing without SSL, but need this to go to prod.

Answer 1

The TLS certificate setup is per listener, so if you want both the listener on port 8883 and the Websocker listener on port 9001 to use TLS then you need to specify the certificates twice eg

listener 8883
protocol mqtt
cafile ./mosquitto/certs/ca.crt
keyfile ./mosquitto/certs/server.key
certfile ./mosquitto/certs/server.crt
require_certificate true
use_identity_as_username true
tls_version tlsv1.2

listener 9001
protocol websockets

allow_anonymous false
cafile ./mosquitto/certs/ca.crt
keyfile ./mosquitto/certs/server.key
certfile ./mosquitto/certs/server.crt


tls_version tlsv1.2

You also need to be very careful about requiring a client certificate for the websocker listener and using it as a the username, because if you are accessing from a Web Browser it is VERY tricky to set up client certificates. Hence I have removed that requirement from the WebSocket listener

Answer 2

You have the listener on port 8883 configured to use websockets; you appear to be connecting using plain MQTT over TCP, not over websockets.

Your client needs to use the same protocol as the broker is configured for on the port the client connects to.

Try commenting out the websockets line in the mosquitto config file, restarting the broker and connecting again.