[BlueOnyx:22648] Re: 5207R/5208R/5209R: Let's Encrypt / ACME related updates released

[Brent Epp]

Hi Michael,

Thanks for the great work here.  We just happened to have a renewal 
scheduled for today and it went off without a hitch.

Just one tiny little thing I noticed: there's a .well-known directory 
inside the site's document root.  Should this still be here?

  - Brent

On 2019-01-27 16:58, Michael Stauber wrote:
> Hi all,
>
> I just published the next set of updates dealing with our "Let's
> Encrypt" debacle and the transition from CertBot to ACME.
>
> The following issues were reported or observed and have now been dealt with:
>
> 1.) Failure to issue/renew certs if the cert includes aliases:
> ===============================================================
>
> This is a long standing issue that we also had with CertBot and before
> we switched to ACME. Yet ACME added another complication, which made the
> issue more obvious.
>
> If you request a cert for www.domain.com the validation is simple. The
> LE client (CertBot or ACME) creates two nested subdirectories and a
> textfile with a validation code: /.well-known/acme-challenge/<UUID>
>
> It then contacts the LE-API and they call the URL up, fetch the
> validation file and check if it matches the expected result.
>
> This usually works. Now throw in "Web Aliases" and it gets more
> complicated. Say you request for www.domain.com, domain.com,
> sub1.domain.com and sub2.domain.com.
>
> This means that separate validation files for the FQDN and *all* aliases
> are created and are polled by the API. If one of the aliases fails to
> get verified, the whole issuing procedure for the LE certificate fails.
>
> There can be several reasons why the validation would fail:
>
> - Vsite has "Web Alias Redirects" (to FQDN) ticked.
> - .htaccess uses mod_rewrite, caching, proxing or forwarding
> - DNS A record for one of the Aliases is wrong or missing
> - Webserver does any kind of redirection.
> - Webserver does some kind of proxying. Some work, some don't.
>
> This list is not exclusive.
>
> The most common issues are "Web Alias Redirects" is ticked or some other
> kind of redirection.
>
> Solution:
> ----------
>
> If you now request an LE certificate *with* aliases and have "Web Alias
> Redirects" ticked for that Vsite, then the handler le_install.pl will
> deactivate that feature automatically before it does the API call for
> the verification from the outside.
>
> The second really common issue are frigging .htaccess files with all
> kind of redirects. Like forced redirects to https://www.domain.com/,
> weird mod_rewrite rules and what not. And neither you or I have much
> control over what a siteAdmin will put into his .htaccess files once you
> allow them to use them in first place. Support wise that's a nightmare.
> We really can't get those users educated not to do that shit, even if we
> tried. And yes, it's possible to code .htaccess files that ignore
> /.well-known/acme-challenge/, but again: People just won't do it,
> because 95% of all people that use .htaccess have no friggin' clue what
> they do there because they've just copied and pasted something they
> found via Google. :p
>
>
> Solution:
> ----------
>
> I added /etc/httpd/conf.d/acme_sh.conf which contains this:
>
> -----------------------------------------------------------------------
> Alias /.well-known/acme-challenge/ /home/.acme/
> <Directory "/home/.acme/">
>      Options FollowSymLinks
>      AllowOverride None
>      ForceType text/plain
>      RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"
>      order allow,deny
>      allow from all
>      <IfVersion >= 2.4>
>          Require all granted
>      </IfVersion>
>      <IfVersion < 2.4>
>          order allow,deny
>          allow from all
>      </IfVersion>
> </Directory>
> -----------------------------------------------------------------------
>
> Means:
>
> Every time an URL like http(s)://<IP|Domain>/.well-known/acme-challenge/
> is called on your server, then it will not be answered out of the
> webspace of the Vsite in question.
>
> Instead this new Apache Alias now points that request to /home/.acme/
> instead. It either returns the Let's Encrypt validation file or triggers
> a 404 error. Attempts of directory listing are denied and trigger a 403
> error instead.
>
> That way we bypass the whole .htaccess related shenanigans on all Let's
> Encrypt related checks.
>
>
> 2.) Logging of ACME transactions:
> =================================
>
> All ACME transactions by GUI or cronjob (issue, re-issue or renewal) are
> now again properly logged (with --debug set to generate sufficient
> details) to /var/log/letsencrypt/letsencrypt.log
>
> If a renewal or cert-issuing fails, check that logfile and it will tell
> you in great detail what went wrong where.
>
> Once I have a sufficiently large base of potential failure messages I'll
> update base-ssl again so that the GUI can show you just the relevant
> bits of the error report. At this time I tries to show you all of it,
> for which simply isn't enough space in the red error-thingy.
>
>
> 3.) ACME issues on EL6 (5207R/5208R):
> ======================================
>
> On EL6 the /.well-known/acme-challenge/<UUID> verification files were
> created with "640" permissions, which proved to be insufficient to make
> them accessible from the outside. In the first release of
> blueonyx-le-acme I already had found all but one instance where ACME
> created these verification files and had throw an "chmod 644" at the
> file right after its creation.
>
> However: I had missed one and that caused renewals not to work correctly
> for Aliases. This has now been fixed as well.
>
>
> These updates for 5207R/5208R and 5209R are now available in the
> BlueOnyx YUM repositories.
>
> This might yet again rock the boat a little by causing Apache restarts
> after the YUM updates (due to throwing /etc/httpd/conf.d/acme_sh.conf
> into the mix) and if that happens: My apologies. I tried to be really
> smooth with that by issuing a "killall -HUP httpd" *and* running Swatch
> afterwards, but you know how it goes: There will be a handful servers
> where this won't do the trick. Like if they have run out of Semaphores
> or just generally have a bad day ...
>
> But: The net result should be a dramatically improved reliability for
> all Let's Encrypt related transactions.
>
>