[BlueOnyx:25607] Re: BlueOnyx 5211R development - progress report

[Michael Stauber]

Hi all,

At the beginning of this month (3rd September) I had posted this 
detailed update about the BlueOnyx 5211R development:

https://www.blueonyx.it/news/308/51/BlueOnyx-5211R-Development-Diary/

I'd like to follow up on that with a further progress report, which you 
can also read here:

https://www.blueonyx.it/news/309/15/BlueOnyx-5211R-Development-Diary/


State of affairs for BlueOnyx 5211R:
=====================================

AdmServ and PHP:
-----------------

AdmServ now uses PHP-8.1.10 via PHP-FPM and the PHP-8.1.10 is installed 
under /home/solarspeed/admserv-php/ and is therefore independent of the 
OS provided PHP version. This version of PHP-8.1 ships with the IonCube 
Loader for PHP-8.1.

During the lifecycle of BlueOnyx 5211R this PHP will be constantly 
updated by us to the latest PHP version.


PHP Framework of the GUI:
--------------------------

The GUI uses the latest CodeIgniter v4.2.5, which we will also keep 
current throughout  the lifecycle of BlueOnyx 5211R as far as required 
to maintain current with the used PHP version.


State of dependencies and software packages:
---------------------------------------------

The following functionality from BlueOnyx 5210R will be absent on 
BlueOnyx 5211R:

- Mailman (explained at the link above)
- Z-Push Active Sync (explained at the link above)
- Web Based File manager

Mailman isn't available for EL9 and my attempts to build it from scratch 
haven't been met with success. So I set this aside for a later time.

Z-Push Active Sync and the Net2FTP integration we had in the BlueOnyx 
5210R GUI are now dead projects and won't work on modern PHP such as 
PHP-8.0 or PHP-8.1. So these have been dropped.

A web based filemanager via the GUI will most likely be added later on 
again, as I have some ideas how we can fill that gap.

Other than that? I do have 90% of all odds and sods sorted by now, so 
via "cceclient" almost the full spectrum of functionality is available 
at this time: Creating, managing and deleting Vsites, Users and all 
other functions one would expect from a working BlueOnyx. The remaining 
10% will be addressed during the completion of the work on the GUI itself.


GUI for BlueOnyx 5211R:
------------------------

As mentioned earlier: CodeIgniter 4.2.5 is radically different from any 
CodeIgniter version that BlueOnyx used before. This means that there is 
no way to do a straight upgrade and continue with existing code. 
Basically everything has do be done at least *slightly* different.

So I started from scratch and solved a metric f-ton of systemic issues. 
Beginning with writing my own dynamic URL-to-Class-loading routing 
mechanism, modified the CSRF mechanism in CodeIgniter 4 to match our 
architecture, stomped hard onto their input verification and filtering 
system to integrate our own and did many other architectural adjustments 
to keep a resemblance of our previous code structure and especially the 
core of our GUI functionality alive.

Then I started small and eventually got the BlueOnyx login page up and 
working, the /swupdate/news page, all Active Monitor pages, all 
base-backupcontrol and all base-console pages.

Initial progress was slow, as each new page I ported had many 
dependencies and used libraries or functions that needed adapting to 
PHP-8.1 and/or the new internals of CodeIgniter. By now the process of 
porting a single old GUI page to a new one is down to 45-60 minutes 
(including testing), 90 minutes if it's a really complex page with lots 
of options.

While working on the porting of the old GUI pages to the new CodeIgniter 
I also made full use of the extended debugging and code runtime 
profiling that is now possible if CodeIgniter runs in development mode. 
This allowed me to identify some horrific performance bottlenecks, which 
I then directly addressed.

Which brings me to the next point:


GUI PERFORMANCE:
================

AdmServ is now using HTTP/2 out of the box, which delivers content in a 
parallelized fashion. Means: That is already faster out of the box.

Usage of PHP-FPM and PHP-8.1 also speeds things up considerably compared 
to PHP-7.2 and DSO like one BlueOnyx 5211R.

The new CodeIgniter 4 and the elimination of some bottlenecks and 
redundancies in the new GUI also speeds things up considerably.

BUT: The biggest speed boost was yet to come and that's a little 
something I hacked together on the sides: CCE-Caching.

When you load a single GUI page, this page also loads two dynamic GUI 
pages as dependencies for localization of error messages during input 
validation as well as the error messages provided by jQuery.

That runs three instances of the GUI-code for a single (visible) GUI 
page. Each of these wants (or needs) to poll CCE for certain CODB objects:

"System" Object
"User" Object of the logged in GUI user
"Capabilities" Object and its 40 NameSpaces for your access rights
"Active Monitor" Object and all its NameSpaces for system faults

Plus whatever CODB Objects the GUI pages needs to display the *actual* 
information you want/need to perform the desired administrative function(s).

Usually these objects are fetched via $cce->getObject("Class"). Which 
runs a "FIND Class" against CODB and finally a "GET <OIDs>". In itself 
that are already three CCE transactions for a single "give me data!" task.

Means: Even if we do nothing else but load a GUI page, we do have 12 
CCEd transactions, of which 2/3rds are repetitive.


Welcome to CODB Caching \o/
============================

We do have single instance CODB Objects, which rarely change during the 
lifetime of a GUI session. And coincidentally and conveniently these are 
also exactly those that we use the most. Even if they change? Then we 
can deal with that easily and can STILL cache them.

So that's what I did.

The GUI now uses the PHP session data as storage for those CODB objects 
that are mostly static and are most used during your GUI session. If the 
data is already in the session-data Cache, CCEd will not be asked, but 
instead the data will be fetched from the cache.

Additionally and index of CODB OIDs will be kept in the session-data 
cache, so that we can skip redundant "FIND" transactions against CODB, 
because we already know what the Object ID of the desired Class is.

Say you login and land on the start page of the GUI: /swupdate/news

During the initial login (before I implemented the Cache) around 56 CCEd 
transactions had to happen. If you then simply reloaded the 
/swupdate/news while remaining logged in? Another 20 CCEd transactions 
had to happen.

Now with the Cache around 16 CCEd transactions happen during initial 
login. From seeing the login page to landing on /swupdate/news, as 
anything important and cache-able will be fetched only once and then be 
served out of the Cache.

Reloading the page past the login? Only 8 CCEd unavoidable transactions 
are required (3x authentication, 3x ending the connection to CCE, 
getting the data we want to display), as the rest is served straight out 
of the Cache.

How much faster is this?

On my test hardware we save around 20-25 milliseconds per FIND/GET 
transactions. This adds up nicely.

Direct comparison:

Login into 5210R:
-----------------

Load time /login page: 			3.5 seconds
Login and loading /news/swupdate: 	6.8 seconds
Reloading /news/swupdate: 		6.5 seconds


Login into 5211R:
-----------------

Load time /login page: 			2.3 seconds
Login and loading /news/swupdate: 	4.9 seconds
Reloading /news/swupdate: 		3.5 seconds


Now two seconds faster doesn't seem to be worth making a fuzz about, 
right? But in practical terms this means a lot, as now every subsequent 
page load past the login loads noticeably faster and that quickly adds 
up to a much nicer experience.


Remaining work prior to release of the BETA:
=============================================

GUI pages:

I still have plenty of GUI pages to port and test. At current speeds I 
expect I need another three weeks for those.


Packaging:

CodeIgniter 4 has a different directory structure than CodeIgniter 3 and 
also needs slightly different Routes.php and Controllers/*.php files. 
Keep thing simple for eventually backporting the 5211R GUI to 5210R 
(which I might!) I decided to install all GUI related files into 
slightly different folders.

Subsequently I need to modify the SVN file structure for BlueOnyx 
modules and the packaging script slightly to deal with this. That will 
take a few days before packaging and automated RPM building is sorted.


YUM Repository:

It's sorted and up. Most OS related bits and pieces are published. The 
new GUI RPMs will be published when the packaging is sorted.


ISO Building:

Not yet started, but I expect few surprises. I should be able to adapt 
the 5210R ISO building procedure nicely to handle EL9 ISO's as well.


BlueOnyx 5211R release date:
=============================

I don't like to make promises on that. However, the coast is pretty 
clear from here and there are no insurmountable mountains to climb 
anymore. The rest is just nitty-gritty digging in the heels and cranking 
code and procedures out. If nothing unforeseeable happens, I think a 
November release of BlueOnyx 5211R is quite doable.


-- 
With best regards

Michael Stauber