[BlueOnyx:13212] CODB corruption: new repair method available

[Michael Stauber]

Hi all,

This one is mostly for the list archives, but also FYI:

Chris Gebhardt and I recently had a case where the CODB database on a
BlueOnyx was pretty much shot to hell.

Symptoms were:

1.) /usr/sausalito/codb/codb.oids was fully correct and had entries for
all OIDs that really existed.

2.) /usr/sausalito/codb/<OID>/ also contained (for the most part)
correct data.

3.) /usr/sausalito/codb/db.classes was probably corrupted.

4.) With cceclient you could do a "find Vsite" or "find User" and it
would return nothing. Whereas it should have shown around 30-40 Vsites
and around 130 User objects.

5.) As CCE's "find" command was no longer returning valid entries,
Active Monitor and various CCE Constructors were creating new Objects
instead of updating the already existing OID entries.

Something like this is pretty rare. I've seen CODB corruptions like this
only sporadically in the last couple of years. But conventional wisdom
at that point usually was:

Restore from the backups.

Well, there are cases where this is noticed too late and the existing
backups also already have inherited the CODB corruption.	

To solve this I got a bit "creative" and did some scripting. In case you
have a CODB that is shot like this, drop me an email and I can sort you
out. One day I hope to clean this script up enough so that it can be
included in BlueOnyx. Right now it's a little quick and dirty job due to
time constrains. :p

What the script does:

It parses a copy of /usr/sausalito/codb/ and creates a hash-table of all
CODB objects and all data within. From that it extracts all "User"
objects, all "Vsite" objects and all DNS related objects.

It cleans the data up a little and then writes out three separate Perl
Scripts that contain all the CCE "Create" and CCE "Set" transactions to
re-create all Vsites (with their settings), all Users (with their
passwords and settings) and all DNS records.

With that it's possible to create the same Vsites, Users and DNS records
on a new server. However: MySQL data, Site-data and User-data such as
websites and emails still needs to be copied manually.

So this isn't a "run a script and you're good" solution, but still
requires some hands on work. But at least it's no longer a total
train-wreck.

In case you need a fix like that in a bend, drop me an email and I'll
try to get you sorted. Like said: Eventually that repair script will be
cleaned up and will become part of BlueOnyx.

-- 
With best regards

Michael Stauber