[BlueOnyx:25578] Re: Large number of users - 5210R - fix available

[Michael Stauber]

Hi all,

Pedro wrote:
>> I have a site with +/- 5000 users runnig on a 5209R. 
>>
>> With 5210R its almost impossible to manage the site (with 5000 users).
>> It takes several minutes just to open vsiteList in "Site Management" 
>> of the GUI;

Pedro was so kind to allow me access to the server in question and I 
could run some diagnostics on it.

It turns out that with 5000 Users the GUI doesn't only make 5000 GET 
Requests on the User Objects, but also also *another* 3-4 GET Requests 
against each User Object to get some NameSpace data. For the 
configuration of Shell, Email, Subdomains and (if installed) OpenVPN.

That means: If someone has 5000 Users in a single Vsite, then the GUI 
needs to make at least +20.000 GET requests just to populate the User 
list and show in one glance what features which user has enabled.

Better "database-design" (the structuring of the data) could have 
prevented this, but that ship has sailed and we can't fix that w/o 
fundamental changes.

With the slow as hell CCE PHP Class of the GUI those +20000 CCE GET 
requests indeed took way too long to be comfortable. In fact whenever I 
tried the GUI timed out after a few minutes. So all I usually got after 
minutes of waiting was a blank page.

I spent this afternoon hacking together a solution, which is currently 
published into the "BlueOnyx 5210R Testing" YUM repository.

The fix to base-user-* published there externalizes the GUI based CCE 
GET requests for the User Objects (and their NameSpaces) into a Perl 
script, which the GUI calls via a secure CCEWrapper. And that 
Perl-Script spits out a single JSON-encoded output that contains all 
requested Objects and all their NameSpaces in one go.

The GUI parses this output and uses it instead of making its own 
individual GET requests. If this round-trip via Perl fails for whatever 
reason, then the GUI automatically falls back to using the old method 
and fetches the data itself.

Perl's access to CCE is faster than the one we have in PHP. That's where 
the speed increase comes from and in Pedro's case the load time for the 
UserList GUI page with 5235 Users went down to 22-23 seconds.

That is still too slow for my taste, but it should be closer to what it 
used to be on the 5209R, where the Zend API CCE module is still used 
instead of the CCE PHP Class.

Without throwing a cache at this problem I may not be able to speed it 
up any further. And caching the user list will create a heap of other 
problems that I need to carefully consider before making any further 
code changes.

Code changes in SVN: 
https://devel.blueonyx.it/trac/changeset?reponame=&new=4281%40%2F&old=4278%40%2F

-- 
With best regards

Michael Stauber