[BlueOnyx:03323] Re: Suggestions for trashed filesystem?

[Michael Stauber]

Hi Chris,

> - Customer system running BlueQuartz
> - Customer has never backed up
> - Server will not boot - "everything" results in segmentation fault
> - Can bring system up via Linux Rescue, mount the LVM, even chroot it
> - Many commands just result in seg-fault, or other errors.
>
> Ideally, I'd love to cmuExport then load the Export into a new box.
> That would be easy.   But whenever I run cmuExport, cmu starts, creates
> its new directory, then says error "chmod: too few arguments" and quits.
>
> Right now I'm just taking everything in /home/sites/ and throwing it to
> another system real quick.   As best I can tell I'm going to have to
> manually create every site, every DNS entry, every account and alias and
> then manually copy everything back in on a new system.
>
> But as long as I have to wait for everything to copy across to the
> backup system anyhow, I wonder if anybody has any other suggestions?

My answer may come a bit late, but unfortunately I was dealing with another 
(similarly wrecking) emergency yesterday.

In a scenario such as the one you mention I'd usually do this:

1.) Take the original disk out and slave it into an Aventurin{e} box.

2.) Create a new VPS with the same OS as the failed box. In your case one 
would set up a BlueQuartz VPS. Create it and then stop the VPS.

3.) Login to the Aventurin{e} master node through SSH. Say your new VPS has 
the VPSID #1 and the hard disk from the failed box has been mounted under 
/extra on the master node.

That then means that the file area of VPS #1 is reachable from the master node 
at /vz/private/1/. Use "mc", "rsync" or your good old "cp" to copy relevant 
parts of the failed disk to the VPS. We don't need everything. We don't want 
to repair it to 100% (because we probably can't!), but just get it into a good 
enough shape that we can run CMU and do a good export.

What needs to be copied from the failed servers disk:

/var/db (if you used PAM on BlueQuartz!)
/etc/mail
/etc/passwd
/etc/shadow
/etc/group
/etc/proftpd.conf (not really needed, but we'll take it, too)
/var/named/chroot/var/named 
/var/named/chroot/etc/named.conf
/var/lib/mysql (or /home/mysql - depends on where your databases are)
/home/sites
/home/.sites
/home/users
/home/.users

That all goes from /extra to the respective places in /vz/private/1/, which 
contains the file system of the VPS #1.

Like said: That's not good enough to get that VPS into a shape where it would 
be good enough to run all services again. But it'll serve our intended purpose 
for creating a CMU-dump after a few additional steps.

Now the tricky part:

We need to - more or less - merge the GUI from the failed box with the one 
from the VPS. This will cause irrepareable damage to the BlueQuartz GUI of the 
VPS. Like all kinds of weirdness, but it should live long enough to get us 
through CMU.

a) Rename /usr/sausalito/codb inside the VPS to /usr/sausalito/codb.bak

b) Copy /usr/sausalito/codb from the failed disk to 
/vz/private/1/usr/sausalito/codb

At that point we have a BlueQuartz VPS which has the following:

- A working OS that can be fired up.

- Thanks to copying /var/db, /etc/shadow, /etc/passwd and /etc/groups we have 
the user account information and their passwords.

- Thanks to copying /usr/sausalito (and especially CODB!) we have all the data 
that CMU needs in order to be able to do an export of the site and user data.

- Thanks to copying /home/.users, /home/users, /home/.sites and /home/sites we 
have the webspace and mailboxes of the failed box. Which is stuff that we also 
need for the CMUexport.

4.) Start the VPS. From the master node run "vzctl enter 1" to directly login 
into the VPS from the master node. 

5.) Run cmuExport. The BlueQuartz VPS already uses the BlueOnyx CMU for 
BlueQuartz, so no need to manually install it from the PKG.

6.) Use the CMU-dump to re-import the sites and users onto a new BlueOnyx VPS.

7.)  After the BlueOnyx VPS is working and has all the sites and users again, 
use the backup functionality built into Aventurin{e} to do daily remote and/or 
local backups of the entire VPS file area.

That's my procedure from memory. Had to use it a couple of times already and 
typically it works quite well and helps to get even the worst train wreck back 
on track. As long as at least one disk of the failed box can still be mounted 
and the file system on it is in a fairly good shape, it usually can be 
salvaged this way.

-- 
With best regards

Michael Stauber