[BlueOnyx:18996] Re: 5208R - Curious behavior of cced - fix is in the making

[Michael Stauber]

Hi all,

Colin wrote:
> Still seeing this. Restarting CCED fixes it briefly but it returns fairly soon again.
> I have just restarted CCED to get into a 5208 machine. Created a new VSite and when it refreshed ... same old! :(
> 
> CCED restart and back in again.
> 
> Not using the cache at the moment.

Yeah. I also just had a 5208R of mine act up. Memcache was since long
disabled. But still it exhibited a number or related failures:

- GUI not working in browser unless you restart CCEd and/or admserv.

- Active Monitor sending Emails in languages other than English (or the
defined language). In most cases these emails were in Spanish. That
should never be the case, as we do have a fallback in Swatch that
defaults to 'en_US' if the language cannot be determined.

- Many cced child processes around, making cced and GUI unresponsive.

- Random failures of cronjobs that need cced. Such as these:

/etc/cron.hourly/newlinq_cron.pl:
Connect to AdmServ failed

Or this one:

---------------------------------------------------------
Subject: 5208r.smd.net: Cambios de estado de monitor activos
From: root
Date 14.01.2016 12:04
To: Vsite <-- WTF?!!

. Active Monitor ha detectado cambios recientes en el estado de su
dispositivo de servidor
Para obtener más detalles, consulte la sección Monitor activo del
Escritorio Server

Summary de cambios.:

* Soporte / Mantenimiento Renovaciones son actuales
---------------------------------------------------------

Usage of "cceclient" showed the root cause of the problem:

--------------------------------------------------
[root at 5208r tmp]# /usr/sausalito/bin/cceclient
100 CSCP/0.99
200 READY
find System
104 OBJECT 3
201 OK
get 3
301 UNKNOWN CLASS System
401 FAIL
--------------------------------------------------

So even with "memcached" disabled we're getting sporadic and
reproducible failures all across the board. Which result in CCEd not
responding in an expected fashion to valid GET requests.

I'm currently preparing a rollback. Which will bring back the (working)
sausalito-cce* we had before the memcache update. This update will also
uninstall the base-memcache module for now.

Now this update is a bit complicated, so it'll take me a couple of hours
to roll back the code and still splice in the "good fixes" we had for
other stuff since then.

I expect to have it ready in 3-4 hours and will post a notice to the
list once I'm done.


-- 
With best regards

Michael Stauber