[BlueOnyx:17069] Re: BlueOnyx 5209R - speed difference explained

[Michael Stauber]

Hi Ernie,

> I am finding the 5209R GUI to be a lot slower than 5207R for example
> clicking on the Server Management tab when you are in the Site Management
> can take 9-10secs when I try the same thing in 5207R it's almost
> instantaneous, like udner a second. Any idea why this mignt be so?

Yeah, that is a good observation and I can confirm that. I also know the
reason for that.

5207R/5208R use the same code for the GUI. 5209R uses around 95% of that
code and the only differences (on the PHP, Perl and Shell-Scripting side
of things) are OS specific differences. Such as Apache 2.4 instead of
Apache 2.2. So they have a different base-admserv, base-apache and a
couple of other small variations. The brunt of the GUI modules is simply
1:1 identical, or will be once I backport some improvements from 5209R
back to 5207R/5209R.

The core of the GUI (alpine.mod) is the same for 5207R/5208R/5209R,
although the one for 5209R is a bit newer (more bugfixes) and I haven't
yet released all those fixes for 5207R/5208R. I'd rather find and fix
all "new" bugs in 5209R before "spreading" them "down" to 5207R/5208R.

So why is 5209R slower? The long answer is kinda hidden in plain sight here:

http://www.blueonyx.it/index.php?mact=News,cntnt01,detail,0&cntnt01articleid=178&cntnt01returnid=54

The Sausalito and Chorizo GUI need two PHP Zend API modules in order to
provide PHP with two core abilities in library form:

a.) cce.so: Provides PHP with the ability to communicate with CCEd.

b.) i18n.so: Provides PHP with i18n support for localization of the GUI.
This is basically our support for the GUI in different languages.

Only the PHP of AdmServ have these two *.so files loaded. This is also
why "the GUI breaks" if you upgrade PHP on BlueOnyx. Because these two
modules need to be compiled against the PHP version that's on the box or
they won't work.

These two PHP Zend API modules go back to the Qube 3 and were initially
written for PHP-3.0. The Qube 3 and RaQ550 shipped with PHP-3.0.6 (if I
recall correctly - it's been a while!), but the development probably
started for a slightly older version of PHP than that.

For TLAS and BlueQuartz they were yet again ported to newer PHP version.
We also ported them for each version of BlueOnyx on new major OS's
versions. Such as for the PHP on EL5 and EL6.

Now with 5209R on EL7 we were facing yet again a new PHP version:
PHP-5.4.16. And that's where I hit a brick wall head on.

Between PHP-5.3 and PHP-5.4 there was a *major* API change in how these
PHP modules work. Porting cce.so and i18n.so was no longer a trivial
"fix something small here and there", but would require some pretty
massive changes that are way outside of the scope of my abilities.

When we reached that point it was clear: There would be no BlueOnyx on
EL7. Without those essential modules? No friggin' chance!

Sooooo ... Steven Howes asked the question on the developer list if it
wouldn't be possible to replicate that functionality in native PHP code
instead. Which was a pretty clever idea - still is, given that we have
no other choice.

And yes: It turned out to be possible.

That's why the alpine module of BlueOnyx 520XR now has this two extra
libraries:

CCE.php:
http://devel.blueonyx.it/trac/browser/BlueOnyx/5209R/platform/alpine.mod/ci/application/libraries/CCE.php

I18nNative.php:
http://devel.blueonyx.it/trac/browser/BlueOnyx/5209R/platform/alpine.mod/ci/application/libraries/I18nNative.php

It works like this: If the PHP that AdmServ uses has the PHP Zend API
modules cce.so and i18n.so loaded (like on 5207R or 5208R) the GUI will
use them instead.

If these two modules are not loaded (like on 5209R where we don't have
them) it automatically falls back to using CCE.php and I18nNative.php
instead. Which are coded in plain and simple PHP.

Both provide Sausalito/Chorizo with the same functionality that the *.so
modules for PHP did: The ability to "talk" with CCEd and the i18n
language support for the internationalization of the GUI.

But of course you can already smell the catch:

The PHP Zend API modules are *much* faster, because they are coded in C
and are compiled binaries.

The PHP libraries on the other hand? They're much, much slower, as they
are loaded and "compiled" at runtime by the PHP interpreter whenever
they're used.

Just to compare: Here is the cce.c code for the cce.so module for EL6:

http://devel.blueonyx.it/trac/browser/BlueOnyx/5207R/utils/cce/client/php/src/cce.c

With comments and spacing it's around 949 lines of C code.

Here is the CCE.php "native PHP" library:

http://devel.blueonyx.it/trac/browser/BlueOnyx/5209R/platform/alpine.mod/ci/application/libraries/CCE.php

With comments and spacing it's not much longer. Yet it is a hell of a
lot more complicated code. Simply for the fact that the Zend API allows
persistent connections, where direct Unix socket access via PHP doesn't
allow us that luxury. At least not when we need asynchronous access to
CCEd, which the GUI needs.

This also boils down to a sad fact that CCE.php does something that the
cce.so doesn't need to do:

For every transaction with CCEd we need to authenticate, send the
command, parse the result and send a "BYE" to terminate the transaction.
The cce.so only needs to authenticate once on page load and at the end
of the page processing it sent a single "BYE". So our CCE.php is about
3x more "talkative" with CCEd than the cce.so - because there is no
other way how this could be done in "native" PHP.

Back in November 2014 I had the initial draft of the CCE.php library
hacked hacked together in just 3-4 days. I then did some speed-tests on
a 5208R to evaluate the differences in speed. That was before later
optimization and fixes. Once with cce.so loaded, so that the GUI used
the cce.so library and once with it disabled, so that it fell back to
using the CCE.php library instead:

How the tests were done:

- Enable/disable cce.so in /etc/admserv/php.ini and restart Apache.
- Load a GUI page and check the render time at the bottom of the page.

/user/userAdd?group=site2:
PHP Module: 0.4519 seconds
PHP Class:  1.1456 seconds

/vsite/vsiteList:
PHP Module: 0.3913 seconds
PHP Class:  1.1366 seconds

/am/amStatus:
PHP Module: 0.4879 seconds
PHP Class:  1.3226 seconds

Net result: The "new" method of using the PHP Class to talk directly to
the Unix socket is roughly 1 second slower on page loads. While 1 second
doesn't sound like much: In reality the PHP Class is 3x to 4x slower
than the PHP module.

So that was the speed test for the cce.so vs CCE.php method. It boiled
down to 0.7-1.1 second speed difference. Which I deemed acceptable (but
not particularly welcome). But the alternative would have been: No
BlueOnyx 5209R.

Then I also ported the other "missing" module (i18n.so) to "native PHP".
I got that working as well, but at a heavy price in performance. While
CCE.php just added 1 second to the load time, I18nNative.php added
another second to that. And that's where it really gets noticeable.

Part of the problem with I18nNative.php are the disk accesses that are
needed to read the respective locale files from
/usr/share/locale/*/LC_MESSAGES/*.mo, which adds a noticeable IO related
delay.

Especially if a GUI pages uses locales out of different files at the
same time. There is also a particular function that checks which locale
files are actually available for all languages. Which means a large
number of (locale related) files and folders are "scanned" to see what's
there. That generates the brunt of the IO related delays, but
fortunately it's only used once on logins. But that also explains why
the logins on 5209R are considerably slower than on 5207R/5208R, too.

As is, the 5209R CCE.php and I18nNative.php "just work", but in the
somewhat longer run I have two goals as far as the CCE and i18n support go:

Optimize the compromise:
=========================

Improve the speed of the "native PHP" classes. Especially the i18n
implementation would benefit from caching the locales to prevent the
heavy IO usage. But both PHP classes also have some room for
optimization to cut down on processing time. There are some regular
expressions in it which are "costly" in terms of processing and CPU
cycles. OTOH there are some loops that could be avoided if we had more
complex and more clever regular expressions instead.

Most ideal solution:
=====================

Find someone who knows enough about the PHP module API and helps us to
get them ported from PHP-5.3 to PHP-5.4. Because as PHP API modules this
"stuff" is *much* faster.

Sooo ... there we have it. In all detail and covering the if's and why's.


-- 
With best regards

Michael Stauber