[BlueOnyx:05517] Re: Cced hangs, admsrv & httpd down @ 4am

[Michael Stauber]

Hi Rickard,

> I've seen the same the last couple of days, while doing a lot of updates
> (packages I'm rewriting). It seems that yum update started from the GUI
> actually kills cced while trying to make a restart of the process.
> 
> This shouldn't happen if updates are done in the background. As I see
> cced shouldn't be restarted if the rpm-packages are installed from the
> GUI. Yum should be started with "at" instead of run from sausalito to be
> safe. We should also disable the server desktop while yum is running.
> 
> Michael, what do you think?

It's a long story and a known issue, yes. Using "at" is one way to solve it, 
but in the longer haul we're probably better off using smarter CCEd restarts.

As you know: Some GUI modules have new or updated Schema files. Which then 
require a CCEd restart in order for things to work correctly. Otherwise saving 
data in the GUI could result in a "XXX is an unknown data type" error message.

Back from the days of old most GUI modules have a CCEd restart command in 
their RPM %post install section. They simply check if the CCEd socket is 
present - and if so, CCEd is restarted. Which is too unscrupulous.

This generally isn't a problem if packages are only installed through the YUM 
cronjob that runs daily. But if such RPMs are installed through "Software 
Updates" or Greg's modifield "BlueLinQ", or as PKGs or through any means 
involving the GUI, then the installation will not go through as you can't use 
the GUI while CCEd is in the process of being restarted.

Which isn't that bad in case of a GUI launched YUM update, as the skipped RPMs 
are then installed through the YUM related cronjob next time around. For 
BlueLinQ or PKG related installs this is deadly, though, as they'll never be 
able to get through. Likewise: You cannot restart AdmServ in the %post section 
of an RPM - if said RPM is installed through any GUI related method.

To get entirely around this only a couple of small design changes are 
required:

a) We as packagers must refrain from blindly restarting CCEd in the %post 
section of the rpmdefs template. Instead we include a check that tests if this 
is an install using the GUI (in which case we don't restart CCEd), or if it's 
a YUM based install (in which case we can safely restart CCEd). 

b) If it's a GUI based install, set a trigger instead. This trigger is then 
picked up by a cronjob that runs periodically and checks if the GUI based "yum 
update" is finished and if the trigger is set. If it is, CCEd is restarted 
once and the trigger is reset.

Example from a (modified) %post section in rpmdefs.tmpl:

# Restart CCEd unless we're updating through the GUI:
if [ ! -f /tmp/yum.updating ]; then
    /etc/init.d/cced.init restart
else
    # Short of restarting CCE now we set a trigger and let a cronjob handle
    # the CCEd restart later on:
    [...]
fi

The Cronjob should check for two triggers: One that says "restart CCEd" and 
one that says "restart AdmServ". If the install is finished and any of the two 
triggers are set, the corresponding service is restarted and the triggers are 
reset. If the GUI based install is not yet finished, the cronjon does nothing 
this time around, which gives the GUI based install more time to finish.

It's on my list to add this to all BlueOnyx modules in the 5107R build and to 
eventually backport it to 5106R as well when we consolidate the code trees 
again and merge the changes back.

-- 
With best regards

Michael Stauber