[BlueOnyx:25253] Re: BlueOnyx 5211R development - commentary

[Michael Stauber]

Hi all,

I'd like to give some more running commentary about the ongoing BlueOnyx
5211R development, which is currently being done on RedHat 9 Beta.

Today I reached the milestone that the GUI is up and is serving the
first page(s) without directly throwing a heap of errors. \o/

Attached is a screenshot from the GUI based setup wizard.

To this date 874 RPMs have been built. 218 SRPMs from other sources than
Red Hat 9 Beta had to be compiled into RPMs. These were mostly taken
from either the Fedora Core 35 code base or (where applicable) SRPMs
from BlueOnyx 5210R were re-built for 5211R.

There is still a ton of things to do and as of now pretty much nothing
works as it should. Which is expected at this early stage of the
development.

The RedHat 9 Beta had started out using OpenSSL 3.0.0-beta3, which in
itself caused tons of problems with anything that needed OpenSSL. A
recent OS update finally switched to the now available OpenSSL 3.0.0
"production ready" release. But there are still a few places where this
is causing transient issues.

Right now neither "ntp" or "ntpsec" are available, so we can't run an
NTP server yet. Both of these don't yet compile against either the new
GLIBC that RHEL9 uses ("ntp") or they have issues with the OpenSSL
("ntpsec"). This will surely sort itself out over time.

The PHP-8 that ships with RHEL9 Beta *still* doesn't have "php-imap",
although the SRPM has provisions to build it. I have successfully
rebuilt the RHEL9 PHP with "php-imap", but packaging that alongside an
OS supplied PHP isn't wise unless I change the name of the resulting
"php-imap" RPM to something that doesn't cause conflicts with possible
later PHP updates. And that will then clash anyway if the OS eventually
ships a PHP with IMAP support.

AdmServ now has a proper Systemd Unit-File and uses it's own separate
PHP-FPM daemon and FPM-pool. Which means that AdmServ now also can use
HTTP/2 for a little speed benefit.

Postfix is up and running, ProFTPd is present in the latest version,
Sendmail (if enabled) works, Apache, PHP and MariaDB work.

The PHP-implementation needs some further thought and work - as
mentioned in earlier messages. But that's something for later on.

Right now I'm still chasing some dependencies to get the
"base-sitestats" and "base-vsite" running. The integration of
"phpMyAdmin" and "phpSysInfo" needs newer components that will work with
PHP-8.

And speaking of that: How does the GUI cope with being lifted from
PHP-7.2 to PHP-8.0?

That is still a matter that needs further investigation and most likely
a ton of fixes. The GUI currently uses the CodeIgniter framework and to
be more precise: v3.1.11. A CodeIgniter v4.1.5 is available and my first
impulse was to use the chance to upgrade the GUI to it.

After reading the documentation and doing some tests I quickly abandoned
that for now. This is something that eventually needs to be done, but
there are so many fundamental changes between CodeIgniter v3.1 and v4.1
that it would result in a complete rewrite of the GUI - from the ground
up. To the point where even future GUI modules and PKGs would need to
contain GUI pages in both the old and new CodeIgniter. That is: If these
modules and PKGs are designed to work on more than one BlueOnyx-model.
Something like we had for a while for 5107R <-> 5207R and 5108R <-> 5208R.

As the v3.1 branch of CodeIgniter still receives security fixes I'm
inclined to keep it for 5211R and push an upgrade of the underlying
framework further into the future.

As is CodeIgniter v3.1 seems to work with PHP-8.0, although I've seen a
few snippets of code already that will need some changes to make these
fully compatible. But it's not as bad as going from PHP-5.4 to 7.2 like
last time around when the GUI was ported from 5209R to 5210R.

Another grey area are currently ProFTPd and Mailman. The latest ProFTPd
works, but changes in the configs require some fixes in the GUI parts
that update the configuration. ProFTPd can now do SNI and serve Vsite's
SSL certificates for connections. Which is nice to have and time
permitting I'll add SNI support to our ProFTPd-integration.

Mailman so far used Python v2 for all the management scripts and RHEL9
ditched Python v2 for good, making Python v3 the only available Python
interpreter. Fedora Core 35 has a "mailman3" that uses Python v3
scripts, but I have the suspicion it's not entirely compatible with the
provisions we have in the GUI. That needs further work.

All in all I see no insurmountable obstacles and all things considered
the port of BlueOnyx to the new upcoming EL9 is progressing nicely.

HOWEVER:

The more I dig into this, the more I'm getting a little upset with
RedHat. Don't get me wrong: RHEL9 Beta is already in a *much* better
shape and running a lot smoother than the RHEL8 Beta was at the same
stage. There also aren't that many critical and fundamental changes
between RHEL8 and RHEL9 like the switch from YUM to DNF and introducing
Streams.

Where it croaks a little it's just due to library changes (GLIBC, Python
3, OpenSSL 3.0.0) and these issues are comparably minor.

At the same time that the RHEL9 Beta was released they also released
Fedora Core 35. Under the hood they are much alike and SRPMs from FC35
can be rebuilt on RHEL9. The FC35 repositories are chock full of almost
all the goodies one could wish for. At the same time the RHEL9 YUM
repositories are as bare as the shelves in a socialist shopping mall.

And that's why I'm upset. The free FC35 has all the bells and whistles,
but the commercial RHEL9 is stripped down to the bones in a fashion that
it's hurting.

Even fundamental building blocks of the OS are missing on RHEL9 Beta at
this time. Such as "cpanspec" and "mock". Both are pretty indispensable
for serious OS integration work and when you need to build/rebuild a lot
of stuff. If you really want to piss off developers, then this is
exactly how to do it. :-/

For three bloody weeks I've been chasing dependencies to get "cpanspec"
(*very* useful for building RPMs of Perl modules) and "mock" (*very*
useful for building RPMs) to work. And there an idiosyncrasy of FC35 bit
me in the ass:

Oh, they have everything I need to build "cpanspec" and "mock". And the
dependencies for the dependencies that their dependencies have. I've
chased down 200 RPMs deep into the dependency hole for both "cpanspec"
and "mock" and I am *still* not there.

Why? Because the Fedora guys build their modules and libraries with
everything *including* the kitchen sink enabled. A module has the
*option* to display time and date in the Mayan calender format? Hey,
let's compile it against the library that can convert to and from the
Mayan calendar! Oh and that module then has some other exotic
dependencies and down the rabbit hole you go, chasing them all. Whether
it's *actually* needed and useful or not.

Some examples:

For "cpanspec" I'm now really close. I only need to fulfill nine more
dependencies. Interestingly these nine dependencies each have only one
or two further dependencies each. That's about 14 RPMs to build. Give or
take.

The second to last final two are perl(Crypt::Blowfish) and
perl(Crypt::CBC). They are light on dependencies and *just* need
perl(Crypt::PBKDF2) and perl(Type::Tiny). Should be easy, right?

Not so much. Because perl(Type::Tiny) has a further nine dependencies,
including perl(MouseX::Types), which even has circular dependencies.

perl(MouseX::Types) needs perl(Any::Moose) and perl(Any::Moose) needs
perl(MouseX::Types) to begin with. You can't build one without the
other. \o/

Using "mock" could solve this nicely, but guess what? It not only needs
a fuckton of Python3 modules that RHEL9 didn't have and I needed to
build first. It needs MouseX as well. /facepalm

Even if I had that: Mock requires python(Virtualenv), which (again) has
python3-pytest-mock as requirement, which needs Mock to begin with <sigh>.

The way how you fix this "on foot" is to install the required
dependencies manually into a contained environment. Installing
*anything* manually (not using an RPM) "taints" a developer box and
makes it less and less transparent if the production OS will have all
the required dependencies as RPMs in the repositories.

One way around that is to manually install the stuff into a users home
directory or into a virtual environment. And then compile against that
to first build an RPM that satisfies one requirement and then you build
the remaining RPM provides the other circular requirement. This will
require a bit of tinkering, but eventually I'll get to that. I'll just
set up a second RHEL9 Beta VPS and shoot that to hell by installing the
required dependencies "on foot", not using (S)RPMs. Some Perl modules
and Python modules (however) *cannot* be installed into users home
directories without purpose defeating modifications.

So while it is a waste of a good VPS I'll probably have to sacrifice one
for this unique purpose. I can't wait until there is an EPEL for EL9.

In many ways I now have RPMs in the "os" YUM repository of 5211R which
eventually will also be available in some shape and form on EPEL, as
RedHat won't bother with them:
http://devel.blueonyx.it/pub/BlueOnyx/5200R/el9/os/x86_64/RPMS/

By the way: RHEL9 didn't even have ImageMagick, so I had to build that
as well from the FC35 SRPM!

That's it for now. In the next few days I'll dig deeper into getting
more parts of the GUI working and will chase me some more dependencies
on the way to a working 'mock'ery. ;-)

-- 
With best regards

Michael Stauber
-------------- next part --------------
A non-text attachment was scrubbed...
Name: BlueOnyx-5211R-Wizard.png
Type: image/png
Size: 143090 bytes
Desc: not available
URL: <http://mail.blueonyx.it/pipermail/blueonyx/attachments/20211220/4f96e1bd/attachment.png>