How to preoperly prepare a DTLS server in OpenSSL 1.1.1

Question

I am trying to get a DTLS "connection" going using OpenSSL 1.1.1.

I am constantly getting a SSL_ERROR_SYSCALL when trying to run DTLSv1_listen() on the socket.

I use a single AF_INET, DGRAM, UDP socket to receive all incoming data. I assumed I could leave it at that and OpenSSL would take care of determining the sender whenever a datagram is received but I am starting to think I am mistaken.

I have: (error handling omitted for brevity)

SSL_CTX *ctx = SSL_CTX_new(DTLS());
SSL_CTX_use_certificate_file(ctx, "certs/server-cert.pem", SSL_FILETYPE_PEM);
SSL_CTX_use_PrivateKey_file(ctx, "certs/server-key.pem", SSL_FILETYPE_PEM);
SSL_CTX_set_cookie_generate_cb(ctx, generate_cookie);
SSL_CTX_set_cookie_verify_cb(ctx, &verify_cookie);

int fd = socket(AF_INET, SOCK_DGRAM, 0);
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, (const void*) &on, (socklen_t) sizeof(on));
bind(fd, (const struct sockaddr *) &server_addr, sizeof(struct sockaddr_in))

SSL *ssl = SSL_new(ctx);
SSL_set_fd(ssl, fd);
SSL_set_accept_state(ssl);
while(DTLSv1_listen(ssl, (BIO_ADDR *) BIO_get_conn_address(SSL_get_rbio(ssl))) <= 0)
...

As I mentioned, that last line gives me an `SSL_ERROR_SYSCALL'.

errno gives me 0 .

I suspect I'm missing some steps in the CTX configuration but I'm not sure what.

I've been looking through some examples and one in particular caught my eye. It seems to create a new socket whenever it receives a datagram and does a connect() on that socket to the remote address. This seems a bit ridicuous to me as I don't think UDP requires a soket per client AFAIK.

Answer 1

According connect with UDP

In BSD sockets one can do a connect on a UDP socket, but this basically just sets the default destination address for send (instead giving explicitly to send_to).

So the success may depent on the used send function.

Generally you can use a UDP socket for DTLS communication to many peers. That requires some mapping between the "association keys/seqn-numbers" and the other peer's address. Though this is pretty much the same as on the server side, it should not be impossible. However, a lot of TLS-derived implementations don't enabled such UDP specific features and so you may be forced to use separate sockets.

One pitfall will be left anyway: If you want to use SNI /server Name Indication) to access the same physical server using different dns names from the same peer, then you will fail.