Error turnkey linux confconsole dehydrated .well-known/acme-challenge 404 NOT FOUND

Question

Using RamNode's Turnkey Linux distro I noticed my websites have expired certificates and https links are flagged with a "Proceed" dialogue. Looking further into the issue, Turnkey Linux uses confconsole with Let's Encrypt to request new certs. Multiple [virtual] domains requires the user to manually use the dehydrated-wrapper which the cronjob calls daily under /etc/cron.daily/confconsole-dehydrated but results in an error:

/var/log/confconsole/letsencrypt.log

[2019-03-09 05:35:04] dehydrated-wrapper: FATAL: An unexpected service is listening on port 80: nginx:
[2019-03-09 05:35:04] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert & key.
Restarting SSL tunnels: [stopped: /etc/stunnel/stunnel.conf] [Started: /etc/stunnel/stunnel.conf] stunnel.
[2019-03-09 05:35:09] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.
[2019-03-09 05:35:09] cron: ERROR: dehydrated-wrapper exited with a non-zero exit code.
[2019-03-10 05:35:04] cron: /etc/ssl/private/cert.pem has expired or will do so within 30 days. Attempting renewal.

Solution: Update /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper with

REPLACE:

netstat -ltpn | grep ":80 " | head -1 | cut -d/ -f2 | sed -e 's [[:space:]].*$||'

WITH:

netstat -ltpn | grep ":80 " | head -1 | cut -d/ -f2 \
   | sed -e 's|[[:space:]].*$||; s|[^a-zA-Z0-9]||'

like in this commit https://github.com/turnkeylinux/confconsole/commit/d1e61c4767c2148663429d63bc3a42925af8cbcd

Then manually run the cronjob again or wait for tomorrow: /etc/cron.daily/confconsole-dehydrated

[2019-03-31 19:26:45] confconsole.hook.sh: SUCCESS: Cert request successful. Writing cert.pem & cert.key for DOMAIN1 to /etc/ssl/private
[2019-03-31 19:26:52] confconsole.hook.sh: SUCCESS: Cert request successful. Writing cert.pem & cert.key for DOMAIN2 to /etc/ssl/private
[2019-03-31 19:26:59] confconsole.hook.sh: SUCCESS: Cert request successful. Writing cert.pem & cert.key for DOMAIN3 to /etc/ssl/private

Thank you and I hope it saves you some time

Related Link: https://github.com/turnkeylinux/tracker/issues/976

Answer 1

I've answered this question in the Question Post itself, however, here's the answer again for the robots.

Solution: Update /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper with

REPLACE:

netstat -ltpn | grep ":80 " | head -1 | cut -d/ -f2 | sed -e 's [[:space:]].*$||'

WITH:

netstat -ltpn | grep ":80 " | head -1 | cut -d/ -f2 \
   | sed -e 's|[[:space:]].*$||; s|[^a-zA-Z0-9]||'

I could further expand on the solution by discussing the root problem.

Because the line in dehydrated-wrapper had been assigning the WEBSERVER variable to nginx: , dehydrated was unable to stop nginx before requesting and updating the certs. Adding nginx: to the case statement would allow you to stop nginx, but would not be able to start the nginx: process, because it does not exist, to host the .well-known/acme-challenges location, resulting in a 404 when dehydrated tries to verify the https ssl certs are correctly configured.