[BlueOnyx:22644] 5207R/5208R/5209R: Let's Encrypt / ACME related updates released

[Michael Stauber]

Hi all,

I just published the next set of updates dealing with our "Let's
Encrypt" debacle and the transition from CertBot to ACME.

The following issues were reported or observed and have now been dealt with:

1.) Failure to issue/renew certs if the cert includes aliases:
===============================================================

This is a long standing issue that we also had with CertBot and before
we switched to ACME. Yet ACME added another complication, which made the
issue more obvious.

If you request a cert for www.domain.com the validation is simple. The
LE client (CertBot or ACME) creates two nested subdirectories and a
textfile with a validation code: /.well-known/acme-challenge/<UUID>

It then contacts the LE-API and they call the URL up, fetch the
validation file and check if it matches the expected result.

This usually works. Now throw in "Web Aliases" and it gets more
complicated. Say you request for www.domain.com, domain.com,
sub1.domain.com and sub2.domain.com.

This means that separate validation files for the FQDN and *all* aliases
are created and are polled by the API. If one of the aliases fails to
get verified, the whole issuing procedure for the LE certificate fails.

There can be several reasons why the validation would fail:

- Vsite has "Web Alias Redirects" (to FQDN) ticked.
- .htaccess uses mod_rewrite, caching, proxing or forwarding
- DNS A record for one of the Aliases is wrong or missing
- Webserver does any kind of redirection.
- Webserver does some kind of proxying. Some work, some don't.

This list is not exclusive.

The most common issues are "Web Alias Redirects" is ticked or some other
kind of redirection.

Solution:
----------

If you now request an LE certificate *with* aliases and have "Web Alias
Redirects" ticked for that Vsite, then the handler le_install.pl will
deactivate that feature automatically before it does the API call for
the verification from the outside.

The second really common issue are frigging .htaccess files with all
kind of redirects. Like forced redirects to https://www.domain.com/,
weird mod_rewrite rules and what not. And neither you or I have much
control over what a siteAdmin will put into his .htaccess files once you
allow them to use them in first place. Support wise that's a nightmare.
We really can't get those users educated not to do that shit, even if we
tried. And yes, it's possible to code .htaccess files that ignore
/.well-known/acme-challenge/, but again: People just won't do it,
because 95% of all people that use .htaccess have no friggin' clue what
they do there because they've just copied and pasted something they
found via Google. :p


Solution:
----------

I added /etc/httpd/conf.d/acme_sh.conf which contains this:

-----------------------------------------------------------------------
Alias /.well-known/acme-challenge/ /home/.acme/
<Directory "/home/.acme/">
    Options FollowSymLinks
    AllowOverride None
    ForceType text/plain
    RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"
    order allow,deny
    allow from all
    <IfVersion >= 2.4>
        Require all granted
    </IfVersion>
    <IfVersion < 2.4>
        order allow,deny
        allow from all
    </IfVersion>
</Directory>
-----------------------------------------------------------------------

Means:

Every time an URL like http(s)://<IP|Domain>/.well-known/acme-challenge/
is called on your server, then it will not be answered out of the
webspace of the Vsite in question.

Instead this new Apache Alias now points that request to /home/.acme/
instead. It either returns the Let's Encrypt validation file or triggers
a 404 error. Attempts of directory listing are denied and trigger a 403
error instead.

That way we bypass the whole .htaccess related shenanigans on all Let's
Encrypt related checks.


2.) Logging of ACME transactions:
=================================

All ACME transactions by GUI or cronjob (issue, re-issue or renewal) are
now again properly logged (with --debug set to generate sufficient
details) to /var/log/letsencrypt/letsencrypt.log

If a renewal or cert-issuing fails, check that logfile and it will tell
you in great detail what went wrong where.

Once I have a sufficiently large base of potential failure messages I'll
update base-ssl again so that the GUI can show you just the relevant
bits of the error report. At this time I tries to show you all of it,
for which simply isn't enough space in the red error-thingy.


3.) ACME issues on EL6 (5207R/5208R):
======================================

On EL6 the /.well-known/acme-challenge/<UUID> verification files were
created with "640" permissions, which proved to be insufficient to make
them accessible from the outside. In the first release of
blueonyx-le-acme I already had found all but one instance where ACME
created these verification files and had throw an "chmod 644" at the
file right after its creation.

However: I had missed one and that caused renewals not to work correctly
for Aliases. This has now been fixed as well.


These updates for 5207R/5208R and 5209R are now available in the
BlueOnyx YUM repositories.

This might yet again rock the boat a little by causing Apache restarts
after the YUM updates (due to throwing /etc/httpd/conf.d/acme_sh.conf
into the mix) and if that happens: My apologies. I tried to be really
smooth with that by issuing a "killall -HUP httpd" *and* running Swatch
afterwards, but you know how it goes: There will be a handful servers
where this won't do the trick. Like if they have run out of Semaphores
or just generally have a bad day ...

But: The net result should be a dramatically improved reliability for
all Let's Encrypt related transactions.


-- 
With best regards

Michael Stauber