[BlueOnyx:15544] Re: Timeout error

[Michael Stauber]

Hi RC,

> Does anyone esle get this error?
> 
> TIMEOUT at /usr/sausalito/bin/check_cce.pl line 25

That's a mechanism that has recently been added to BlueOnyx to monitor
CCEd and - if need be - to restart it.

Before the Swatch (Active Monitor) cronjob runs, it calls
/usr/sausalito/bin/check_cce.pl. That Perl script uses the Perl Module
Time::Out and tries to establish a CODB connection to check for the
"System" Object.

If CCEd doesn't respond within 5 seconds with the expected reply, then
it is assumed that it is hung up and needs restarting before Swatch is run.

/usr/sausalito/bin/check_cce.pl either responds with "SUCCESS" (all is
good), "FAIL" (CCEd is down) or "TIMEOUT" (CCEd is not responding within
5 seconds).

In case of FAIL or TIMEOUT CCEd will be killed off an will be restarted
automatically.

This mechanism is also used from within the GUI. In the past when you
accessed the GUI and CCEd was down or unresponsive, you would see the
red text on white background that informed you that CCEd was down and
should be restarted. Instead of showing that directly it now also uses
check_cce.pl to determine the status of CODB first and - if need be -
restarts it.

Now check_cce.pl is usually called in a fashion that doesn't generate
cron emails and actions it performs are silenced. But if the server is
really overburdened by other tasks, it could be that the muffling of its
output still gets emailed. Such as in the case that you reported.

It's nothing to worry about in itself. If it times out, it'll right
itself within the next 15 minutes when Swatch runs again, or with the
next GUI transaction that is performed.

For technical details:

The script that checks CCEd is this:

/usr/sausalito/bin/check_cce.pl

The script that restarts CCEd is this:

/usr/sausalito/bin/cced_unstuck.sh

It comes together in the Active Monitor shell script that calls Swatch:

/usr/sausalito/sbin/swatch.sh

CCEDUP=`/usr/sausalito/bin/check_cce.pl`
if [ "$CCEDUP" != "SUCCESS" ];then
        /usr/sausalito/bin/cced_unstuck.sh >/dev/null 2>&1
        sleep 5
fi

-- 
With best regards

Michael Stauber