[BlueOnyx:25209] Re: BlueOnyx 5211R development - commentary

[Michael Stauber]

Hi all,

I thought I provide some running commentary on the 5211R development.
This will be a longer email and there are some hard design choices to
make. For which I'm asking for your input.

These are mentioned earlier on in this long message. The rambling
follows after that. :p

Today the development of BlueOnyx 5211R on EL9 and continued where I had
left off a few days ago:

Getting a proper build-chain set up in order to be able to build the
most basic GUI components. Which are these:

base-admserv-capstone
base-admserv-glue
base-alpine
base-sauce-basic
perl-handler-utils
sausalito-cce-client
sausalito-cce-server
sausalito-devel-tools
sausalito-i18n

As usual a metric ton of RPMs needed for the build aren't available on
the RHEL 9 Beta YUM repositories.

To get the basic building blocks of BlueOnyx compiled, I needed to
manually build and provide the following RPMs:

flex & flex-devel
gdbm, gdbm-libs & gdbm-devel
glib, glib-devel
glib-ghash
imake
perl-Jcode
perl-Linux-Pid
perl-TeX-Hyphen
perl-Text-Autoformat
perl-Text-Reform
perl-Time-Out
re2c

This (short) list isn't representative, as there are other aspects that
require a hell of a lot more dependencies to be fulfilled the further I
get down in the build tree. But the above were enough for the basic
building blocks of the GUI.

So by now I have AdmServ up (barely) and CCEd works.

While this sounds encouraging: There are caveats and these will require
some hard design choices.

On EL9 legacy InitV or Upstart start scripts no longer work, as the
required helper scripts are gone for good. Only two services still used
old InitV start scripts:

- admserv
- pam_abl

The pam_abl situation was fixed easily. The functions of that script are
now performed by the Systemd unit file cced.init.service, which starts
up CCEd.

As for AdmServ? Due to complexities of being an Apache fork with special
environment variables AdmServ needs a pretty sophisticated Systemd Unit
file. Right now I'm starting AdmServ manually on my devel box and have
several prototypes of a Systemd Unit scripts for it on the cooker. One
of them will eventually work. :p

Now the kicker: RHEL9 beta has no mod_php. It ships with PHP-8.0.6 and
in the default configuration its Apache uses PHP-FPM. Which isn't bad in
itself.

BUT: They do NOT have the libphp.so of PHP-8.0.6 anywhere in the
repositories.

How it looks on EL8:

[root at 5210r ~]# LC_ALL=C yum whatprovides "*/libphp*"
php-7.2.24-1.module_el8.2.0+313+b04d0a66.x86_64 : PHP scripting language
for creating dynamic web sites
Repo        : @System
Matched from:
Filename    : /usr/lib64/httpd/modules/libphp7.so

How it looks on EL9:

[root at 5211r RPMS]# yum whatprovides "*/libphp*"
Updating Subscription Management repositories.
Error: No Matches found

Great, innit? <sigh>

I get the reasoning behind that: Apache cannot do HTTP/2 if PHP is
incorporated via mod_php (libphp.so). And RedHat chose to sacrifice the
default mod_php configuration for the sake of enabling HTTP/2 by default.

Can we make do with PHP-FPM? Yeah, sure. But that would also mean that
Vsites can no longer have the option to run PHP as DSO or DSO+mod_ruid2.

Means we'd only have PHP-FPM and suPHP. THAT is not good. I am all in
favor of sacrificing HTTP/2 for mod_php, as we can always enable HTTP/2
again (at least for HTTPS) if we use Nginx as SSL proxy. An option that
since long has already existed in 5209R and 5210R.

So this is the first design choice for consideration:

Option #1:
===========

Apache with mod_php (libphp.so)
--------------------------------

Benefits:

We retain all PHP implementations that we already have on 5209R and 5210R:

	- Off
	- DSO
	- DSO+mod_ruid2
	- suPHP
	- PHP-FPM

Downsides:

a) No HTTP/2 unless you use Nginx as SSL proxy and then you only have
HTTP/2 via SSL/TLS.

b) We (or better said: I) need to provide a libphp.so for whatever
version of PHP ships with EL9 and whenever PHP OS updates are released,
I need to ship an updated build. If the libphp.so no longer matches the
rest of the PHP install as provided by the OS, it won't matter much as
far as stability goes, but security wise there could be issues.

Option #2:
===========

Apache with PHP-FPM and HTTP/2
-------------------------------

Benefits:

Closer to stock configuration of EL9, allows HTTP/2 for both HTTP and
HTTPS out of the box.

Downsides:

Only supported PHP options will be PHP-FPM and suPHP. Even with
(optional) Solarspeed PHP from the shop we then could not enable the DSO
variants of PHP without turning HTTP/2 off for good.


It's possible that RedHat gets pestered enough to eventually include a
libphp.so before this goes final. Or that the EPEL guys will pick up the
slack. But we can't count on that *or* delay our design decision until
we know for certain.

This situation is highly uncomfortable and a design choice needs to be
made ASAP. Before final release of 5211R this choice can be changed or
reversed, but once the final release is out, the chosen way has to stay
the way it was on release.


More fun and games and also some rambling:

OpenSSL 3.0
============

EL9 ships with this OpenSSL version:

[root at 5211r ~]# rpm -q openssl
openssl-3.0.0-0.beta2.7.el9.x86_64

I kid you not. 3.0.0-0.beta2. BETA2!!

So everything using OpenSSL is compiled against that.

Again: I get it. Purely  from a design point of view it's the OpenSSL
version that will be in use for years to come. The only alternative
would have been OpenSSL-1.1.1, which is in LTS support. But that expires
in September 2023 and then RedHat would be left holding the bag on
keeping a decrepit OpenSSL-1.1.1 alive if they had chosen to use it for
EL9. Like always RedHat has a shitty timing of picking software for
their Enterprise Linux, as some stuff they picked (both now and in the
past) was just barely mature enough. Like this OpenSSL-3.0.0. BETA2! :p

And indeed: OpenSSL-3.0.0 still has issues. And programs compiled
against it have their own transient issues as well.

Take "subversion" for example, which we use as code repository for
BlueOnyx. On EL9 I was unable to use "svn checkout" against
https://devel.blueonyx.it, because the "svn" that ships with EL9 Beta
has a broken SSL implementation. Our repository was forcing connections
to HTTPS and I had to explicitly turn it off and use HTTP for code
checkouts on EL9. Just debugging that issue (and reconfiguring our SVN)
took me some hours.

We can expect some other surprises in regards to OpenSSL 3.0.0 in EL9
and can only hope that they are resolved before EL9 gets released.


PHP 8.0.6:
===========

Aside from the missing php-8.0.6-module RPM we're also missing the
following RPMs:

php-imap
php-mcrypt
php-embedded (another name for the RPM with the libphp.so)
php-recode
php-xmlrpc

The missing IMAP support is ridiculous. I won't comment any more on
that, because I already ran out of my daily allocation of the word
"fuck". And I'd need several to explain what I think about a missing
php-imap. :p

The only other critical item on the list is "php-mcrypt". But to be
honest: The PHP on EL8 also didn't have that and I provide it via the
BlueOnyx 5210R repository. Might have to do that again.

We need the MCrypt support for anything in the GUI that requires
encoding/decoding, so it's a must have. I'll get it sorted one way or
the other.


My immediate to-do list for 5211R now looks like this:
=======================================================

1.) AdmServ: Create a working Systemd Unit-Script for it.

2.) Try to get AdmServ working with PHP-FPM backend. But that
    PHP-FPM must be a separate instance independent from the
    general PHP-FPM daemon that the OS uses for Apache. Otherwise
    FPM daemon restarts initiated by the GUI would cause the GUI
    to error out. And we can't have that. For just this purpose
    mod_php support would be really nice to have <sigh>.

3.) Upgrade base-alpine's CodeIgniter framework with a version that
    works with PHP-8.0. Upgrading CodeIgniter isn't trivial, as ours
    has some highly customized heavy duty third party modules for the
    internal routing, but it has to be done.

4.) Get MCrypt working in PHP-8.0.6

5.) Fix all mod_proxy calls in AdmServ that currently error out due
    to HTTP/2, which has it's own proxy module.

6.) Make all PHP scripts of the GUI compatible with PHP-8.0. Fortunately
    the changes between PHP-7.2 and PHP-8.0 aren't *that* massive. It'll
    still require quite some effort due to the massive amount of PHP
    libraries, Classes and GUI pages, though. Still: About 80-90% of the
    current PHP errors in the 5211R GUI will go away on their own after
    CodeIgniter has been upgraded to a PHP-8.0 capable version. THAT
    alone makes usage of a PHP framework a great bonus.

7.) Get "cpanspec" working.

    We need around 200-230 Perl modules for BlueOnyx. Some of these
    modules are only needed to *build* other Perl-Modules. A stock
    BlueOnyx 5210R has 317 Perl related RPMs installed.

    My 5210R build box has 552 Perl related RPMs installed. Which
    gives an idea of how many extras I also need to build for 5211R.

    With "cpanspec" I can easily build RPMs of the latest modules
    available in the Perl code repository MCPAN. I tell it the
    module name and it grabs the sources in a tarball and builds
    an RPM Specfile. Which I can then use to build and RPM.

    And it can automatically follow dependencies to download the
    sources and create Specfiles for all dependencies that the build
    of said module has. And the dependencies of their dependencies.
    Which is such a time savior that I could cry.

    Sadly "cpanspec" itself has so many dependencies that it'll take
    days of manual module building before it even leaves the pit.

    But once I have it up and running it'll be a godsend.

8.) Get the rest of the BlueOnyx modules built that make up the GUI.

    Many of these have their own dependencies as well. Like base-email
    needing the stock Sendmail, our modified Postfix and Dovecot. Or
    base-shell needing Shellinabox and base-ftp needing a modified
    ProFTPd. Others need a certain Perl or PECL module and what not.

    These dependencies need to be chased and built.

This will mostly be a matter of dedication and elbow grease, so there is
no need to worry. It's manageable and this is the 8th time I've been
doing a port of BlueOnyx to a new OS or architecture.

Under the hood EL9 holds fewer drastic changes than EL7 -> EL8 did.
We're still using Apache 2.4 and aside from the mod_php, HTTP/2 and
mod_proxy for HTTP/2 we're on the same general turf.

The OS still ships with a working Sendmail (although RedHat had hinted
at deprecating it), so our base-email remains the same. And that means
we can still let the GUI build a Sendmail configuration, which our
modified Postfix parses to build its own configuration. I had been
worried that I had to modify base-mail to directly crank out a Postfix
config. In which case things would have gotten really complicated.

The final deprecation of legacy InitV and Upstart scripts is only a
minor nuisance, as only AdmServ still needs to be ported to Systemd.

That RPMs are served via DNF and through AppStreams? We had that already
tackled in 5210R. Nothing more to do on that front.

So platform wise the changes aren't as massive as last time around,
which should make this a smooth affair.

I'm not putting myself under undue pressure here and the goal is to have
a stable and working copy of 5211R on RedHat Enterprise Linux Beta 9
ready in early January 2022.

I have no idea when RHEL9 will be released, nor when AlmaLinux or
RockyLinux will have their Betas or Release-Candidates of EL9 ready.

But I assume I'm done with 5211R before that and can then tackle any
issues that *might* pop up due to potential differences between RHEL9
Beta and an AlmaLinux or RockyLinux 9 release.

That's all for now.

-- 
With best regards

Michael Stauber