[BlueOnyx:10577] Re: CCED is hanging on startup and is still there after a couple hours.

[jtowne at turtlehut.com]

>> Hi Jeremy,
>>
>>> We were having some issues with adding sites to a 5106R so I tried
>>> restarting CCED and admserv but it kept hanging on CCED.  There were
>>> usually 2 processes started but never gave the OK/done.  So we decided
>>> to reboot the server because someone was going to be in the area today.
>>> Well that was 2.5 hours ago and it is still setting at starting CCED.
>>> All sites are down and we can't shell in.  Any ideas how to proceed?
>>
>> Boot off the BlueOnyx CD in rescue mode. To do so, insert the CD, reboot
>> and
>> at the CD installation menu type "linux rescue" instead of pressing
>> return.
>>
>> It'll ask for keyboard and network settings. You don't really need to
>> set
>> the
>> network, so you can skip that step. It will ask you if you want to mount
>> the
>> hard disks, which you confirm.
>>
>> Once you get the shell, type:
>>
>> chroot /mnt/sysimage/
>>
>> That willl mount the filesystem of the CD as /, so that you can directly
>> work
>> with the installed OS instead of the OS provided by the rescue mode.
>>
>> Type this command next:
>>
>> /sbin/chkconfig --level 2345 cced.init off
>>
>> That will prevent cced.init from starting up during the next reboot.
>>
>> Press CTRL+D twice to exit out of the chroot and to exit the rescue
>> mode.
>> Reboot and while the system goes into reboot remove the CD from the
>> drive
>> and
>> let the server boot up the normal way.
>>
>> The system will come up again, but without starting CCEd. That will
>> prevent
>> the GUI from working and httpd may also not work, but you can at least
>> get
>> back into the box using SSH for troubleshooting the real issue.
>>
>> Most likely the CCEd issue is a corrupted CODB database. To troubleshoot
>> that
>> create the script /root/oid.sh and copy and paste the following code
>> into
>> it
>> (without the ---- lines):
>>
>> --------------------------------------------------------------------------------------
>> #!/bin/bash
>> LAST=-1
>> MIN=-1
>>
>> for X in `ls /usr/sausalito/codb/objects/ | sort -n`
>> do
>>   MYNEXT=$(( $LAST + 1 ))
>>   if [ $MYNEXT -eq $X ]
>>   then
>>     LAST=$X
>>   else
>>     if [ $LAST -ge 1 ]
>>     then
>>       if [ $MIN -eq $LAST ]
>>       then
>>         echo -n $LAST,
>>       else
>>         echo -n $MIN-$LAST,
>>       fi
>>     fi
>>     LAST=$X
>>     MIN=$X
>>   fi
>> done
>> if [ $MYNEXT -lt $X ]
>> then
>>   echo -n $LAST
>> else
>>   echo -n $MIN-$LAST
>> fi
>>
>> echo ""
>> echo "/usr/sausalito/codb/codb.oids reports:"
>> cat /usr/sausalito/codb/codb.oids
>> echo ""
>> --------------------------------------------------------------------------------------
>>
>> Make the script executeable ("chmod 700 /root/oid.sh") and run it. CCEd
>> does
>> NOT have to run for this test.
>>
>> It will generate output similar to this:
>>
>> [root at server ~]# /root/oid.sh
>> 1-51,56-59,65-114,127-135,149-153,160-175,181-201,204-207,219-229,231-257,260-291,328-329,337-347,363-367,377-382,392-397
>> /usr/sausalito/codb/codb.oids reports:
>> 1-51,56-59,65-114,127-135,149-153,160-175,181-201,204-207,219-229,231-257,260-291,328-329,337-347,363-367,377-382,392-397
>>
>> The lines with the numbers are what we are looking for. They MUST be
>> identical. If they are not, then that is the problem.
>>
>> So if the lines are NOT identical, do this:
>>
>> Edit /usr/sausalito/codb/codb.oids and remove everything in it. Then
>> copy
>> the
>> first numbered line into that file.
>>
>> Afterwards restart CCEd with this command:
>>
>> /etc/init.d/cced.init restart
>>
>> That should fix it.
>>
>> --
>> With best regards
>>
>> Michael Stauber
>> _______________________________________________
>> Blueonyx mailing list
>> Blueonyx at mail.blueonyx.it
>> http://mail.blueonyx.it/mailman/listinfo/blueonyx
>>
>
> For some reason when I do this I got the server back up.  Then I fixed the
> OIDS so they matched.  I restarted CCED but it never completed for hours.
> I logged in with another terminal and the OIDS don't match again.  Why is
> this?  What am I doing wrong?
>
> Jeremy
>
>
>
>
> _______________________________________________
> Blueonyx mailing list
> Blueonyx at mail.blueonyx.it
> http://mail.blueonyx.it/mailman/listinfo/blueonyx
>
 To add I just found out that in the administration it shows a site that
doesn't have a folder anymore in /home.  It looks like a failed site
removal or someone removed it incorrectly which could have happened since
there are multiple hands on the cookie jar.