[BlueOnyx:22923] Re: DKIM with underscores - and 5210R progress report

[Michael Stauber]

Hi Colin,

> Will we  be able to do this in 5210R? (

Yes, I added it to the list of features and fixes that will get added to
5210R (and older) once I've got the 5210R code sorted out.

> How is that going?

The development of 5210R is progressing. I'm not saying "progressing
nicely", because it's been anything but nice so far.

We're dragging around some legacy code that goes back to the 90's.
Mostly in the form of Perl scripts. Perl has *always* been really
painless when it came to running older Perl code on newer versions of
Perl. It has rarely bitten us. Now it did.

POSIX(isalpha) has been deprecated and a couple of other things as well.
So a lot of Handlers/Constructors were complaining until I had fixed them.

The GUI itself is a mess on a whole different level. AdmServ now runs
PHP-7.2 and the GUI had code that worked on PHP-5.4 at best. While most
PHP scripts and libraries have been retouched at least once since 2014,
many still use functions and procedures here and there which are now
getting the deprecation hammer.

Just upgrading the underlying CodeIgniter PHP Framework from 2.2 to 3.1
broke all kinds of things and it took me weeks to find and fix the last
of those issues. The issue was mostly stuff that I had hacked into the
old CodeIgniter to extend it with functionality it was missing.
Fortunately I had done it in a modular fashion, so I didn't need to diff
out the changes between a stock CodeIgniter and our modified one. But as
2.2 and 3.1 are internally a bit different the nooks and crannies that
my modules were supposed to fit in there were all slightly different and
that needed finding and fixing.

The real bummer is of course all the BlueOnyx UIFC classes and Libraries
that provide the interface to CCEd, the i18n translation engine and
render our form fields and GUI pages.

Stuff like this doesn't work anymore:

-------------------------------------------------------------------
	$block =& $factory->getPagedBlock();

        $block->addFormField(
            $factory->getRawHTML("userList", $scrollList->toHtml()),
            $factory->getLabel("userList"), $subsId
        );
-------------------------------------------------------------------

That now needs to be this way:

-------------------------------------------------------------------
	$block = $factory->getPagedBlock();

        $x = $factory->getRawHTML("userList", $scrollList->toHtml());
        $block->addFormField(
            $x,
            $factory->getLabel("userList"), $subsId
        );
-------------------------------------------------------------------

All PHP classes also suffered from their class constructor having the
same name as the class. Which is now forbidden.

So this ....

-------------------------------------------------------------------
class ModifyButton extends Button {
  function ModifyButton(&$page, $action) {
    [...]
    $this->Button($page, $action, $type, $ButtonLabel);
  }
}
-------------------------------------------------------------------

... now must be like that:

-------------------------------------------------------------------
class ModifyButton extends Button {
  public function __construct(&$page, $action) {
    [...]
    parent::__construct($page, $action, $type, $ButtonLabel);
  }
}
-------------------------------------------------------------------

These are just tiny changes, but they have to be made as PHP is throwing
deprecation warnings or outright errors otherwise. On pretty much every
friggin' GUI page.

Then there are plenty of wile ... each() functions that need to be
changed, as each() has been deprecated as well.

In the past we used the Zend Extensions 'cce' and 'i18n' for two tasks
of critical importance:

- 'cce' for letting PHP talk to CCE.
- 'i18n' to use the i18n system language files to show the texts in the
  GUI in the various languages.

Both these modules will not build against PHP-7.2 without a coding
effort that's beyond my ability. I'm no C-coder, so I can't do that.

For 5209R I already had substituted native PHP functions that can take
over if the PHP Zend Extensions are absent. However, these two native
PHP classes are a bit brutish and weren't working 100% right all the
time under all conditions. I'm now slowly finding the places where they
go off the rails and fix them one by one.

I'm currently about one 15% done with the GUI pages and about 60-70%
done with the UIFC functions and it'll take me the better part of June
to get them sorted.

Lastly there are the OS related changes where certain functions or
methods we used either went away, got replaced by something else or
where config files of services have drastically changed.

The Network configuration for instance? RHEL8 now exclusively uses
NetworkManager. This has been around in previous versions as well, but
we always disabled it and did the network config "on foot".

Now even "/sbin/ifup" is a wrapper for a NetworkManager function, so we
*have* to use NetworkManager. Which isn't that bad, but required a
metric f***-ton of changes to base-network and base-vsite. Which are
almost done by now, but could use some cleanup.

The newer Dovecot and Proftpd have a different config file format, so I
need to tackle the scripts that write out the new configs. Right now
they produce garbage that lets the services fail.

The Nginx SSL proxy? The Nginx in RHEL8 is fairly OK, but I'll again
substitute my own as I did for 5209R. That hasn't been done yet.

TLSv1.3? All TLS enabled services (Sendmail, FTP, Dovecot, Apache and
Nginx) can use TLSv1.3, but I still need to update the handlers that
update the config files for these services.

All in all: There is still an impressive mountain of work that needs to
get tackled, but I'll get it done.


Fun stuff:
===========

Now here is a hilarious "Oh f***!"-moment I had on Friday and Saturday
and for a moment it looked like I had shot my 5210R development box to
hell and then some, threatening months of progress.

The 5210R devel box is a VPS on Aventurin{e} 6109R. As there are no
RHEL8 OS templates around yet it's not a "container", but a "VM".

OpenVZ 7 realizes these via Quemu and you can create and manipulate them
with the same commands that you use for containers. Just the syntax is
slightly different and there are some things you can do with CTs that
you can't do with VM's.

For example: On the 6109R node I can run "prlctl enter <VPSID>" and get
a shell inside the CT. That *also* works with VM's, but only if the
shell-tools have been installed inside the VM. The shell-tools for RHEL8
aren't done yet.

So I only have three ways to get shell inside the 5210R devel box:

1.) Via SSH. Which obviously requires a working network connection.
2.) Via https://cockpit-project.org/ net console, which is integrated
    into RHEL8 and can be activated. Requires working network
    connection.
3.) Via VNC web console from the 6109R GUI. Works even if the VM has
    lost it's network configuration and also allows me to access the
    Grub2 boot menu and see the startup/shutdown progress.

I was using the GUI to create a user for the first time and the handler
mod_quota.pl barfed with an error message. It had failed to set the
quota. I quickly realized that I had never edited /etc/fstab to turn the
quota on for / or /home.

Now I got a bit cautious there. A small typo and the VM wouldn't boot
anymore. Which would be *really* bad.

So I thought: Now is a good time to do a another backup anyway.

I went to the 6109R shell, stopped the VM and used "prlctl clone" to
create a carbon copy of the 5210R development VM. I have a few of these
copies around by now. Then I started the original VM again in order to
edit /etc/fstab to enable the quota.

But alas: The VM booted without network. So ... no SSH. No GUI. No way
to access it.

Except for the KVM web console. Which usually had worked exceptionally
fine. But now it didn't. It didn't show the usual login screen like this:

---------------------------------------------------
Red Hat Enterprise Linux 8.0 Beta (Ootpa)
Kernel 4.16.0-32.el8.x86_64 on an x86_64

5210R login: ||
---------------------------------------------------

There was no blinking cursor that expected you to type the login name.
And the text (or fragments of it) were garbled all over a grey instead
of a black background with three squares in the middle.

The login screen was still showing parts of the boot splash-graphics and
even pressing ESC or "f" or "s" wouldn't make that go away.

Entering "root" and the root-password in the blind wouldn't let me in
either. Some of the garbled texts seemed to indicate "incorrect password".

At times the "incorrect password" error message even popped up before I
was done typing.

So ... that's a full bunch of problems at once:

- Splash-background didn't go away.
- Possible root password change due to some handler snafu. Might have
  reverted back to the default password of new installs.
- VNC-console problems like sporadically sending enter when I
  wasn't doing so.
- Network config shot to hell.

What about the backups? Each and every one of them had the same issue.
Yay! \o/

The KVM console let me access the Grub menu. But with the usual
problems: German keyboard, English input required. The noVNC console
then also maps some special characters differently, complicating the
typing a little. But it's doable.

I managed to edit the grub boot parameters to be able to boot in "rw"
mode, with "rhgb" eliminated from the parameters (that fixed the splash
screen issue), with "quiet" added (that fixed the error messages popping
up on console) and with "vconsole.keymap=de" to get a German keyboard.
Finally with "init=/bin/bash" to get a root shell without password prompt.

That allowed me to reset the root password. Why was the bloody network
not working? Well ... long story short: The VM had been running since
December by now. With /dev/ens3 as primary network interface. Now after
the first reboot it had finally come up with /dev/eth0 and the
ifcfg-ens3 file needed to be renamed to ifcfg-eth0.

Of course this happened in the middle of the night after a long day. The
"fix" that I posted above? Which takes no more than five minutes to
execute? I won't embarrass myself telling how long it took me as it was
a whole lot longer. And seeded with the irrational fear of possible loss
of three months of progress if I could no longer boot that bloody VM.

That fear was irrational, because "prlctl mount <VPSID>" also works for
VMs on 6109R and makes the disks and their file areas of Quemu VMs
available under /vz/private/<UUID>/ for read/write access. You can even
edit the Grub configuration from there, although it's not recommended.
Editing the contends of a Quemu disk that way will force an FSCK run
when the VM is unmounted and restarted, but it's a good way to get these
things fixed or even rescued off a broken box when in a bend.

I'm *really* glad that I decided to do the development work of 5210R on
an Aventurin{e} 6109R node. Because there I have multiple easy ways at
my disposal to back up the whole VM as well as recover it from all the
expected troubles that it inevitably will run into while it's under
development. Like shedding the network configuration or resetting the
root password back to the default while I work on the setup-wizard and
installer.

-- 
With best regards

Michael Stauber