[BlueOnyx:10463] Re: CCED is hanging on startup and is still there after a couple hours.

[Michael Stauber]

Hi Jeremy,

> We were having some issues with adding sites to a 5106R so I tried
> restarting CCED and admserv but it kept hanging on CCED.  There were
> usually 2 processes started but never gave the OK/done.  So we decided
> to reboot the server because someone was going to be in the area today.
> Well that was 2.5 hours ago and it is still setting at starting CCED.
> All sites are down and we can't shell in.  Any ideas how to proceed?

Boot off the BlueOnyx CD in rescue mode. To do so, insert the CD, reboot and 
at the CD installation menu type "linux rescue" instead of pressing return.

It'll ask for keyboard and network settings. You don't really need to set the 
network, so you can skip that step. It will ask you if you want to mount the 
hard disks, which you confirm.

Once you get the shell, type:

chroot /mnt/sysimage/

That willl mount the filesystem of the CD as /, so that you can directly work 
with the installed OS instead of the OS provided by the rescue mode.

Type this command next:

/sbin/chkconfig --level 2345 cced.init off

That will prevent cced.init from starting up during the next reboot. 

Press CTRL+D twice to exit out of the chroot and to exit the rescue mode. 
Reboot and while the system goes into reboot remove the CD from the drive and 
let the server boot up the normal way.

The system will come up again, but without starting CCEd. That will prevent 
the GUI from working and httpd may also not work, but you can at least get 
back into the box using SSH for troubleshooting the real issue.

Most likely the CCEd issue is a corrupted CODB database. To troubleshoot that 
create the script /root/oid.sh and copy and paste the following code into it 
(without the ---- lines):

--------------------------------------------------------------------------------------
#!/bin/bash
LAST=-1
MIN=-1

for X in `ls /usr/sausalito/codb/objects/ | sort -n`
do
  MYNEXT=$(( $LAST + 1 ))
  if [ $MYNEXT -eq $X ]
  then
    LAST=$X
  else
    if [ $LAST -ge 1 ]
    then
      if [ $MIN -eq $LAST ]
      then
        echo -n $LAST,
      else
        echo -n $MIN-$LAST,
      fi
    fi
    LAST=$X
    MIN=$X
  fi
done
if [ $MYNEXT -lt $X ]
then
  echo -n $LAST
else
  echo -n $MIN-$LAST
fi

echo ""
echo "/usr/sausalito/codb/codb.oids reports:"
cat /usr/sausalito/codb/codb.oids
echo ""
--------------------------------------------------------------------------------------

Make the script executeable ("chmod 700 /root/oid.sh") and run it. CCEd does 
NOT have to run for this test.

It will generate output similar to this:

[root at server ~]# /root/oid.sh 
1-51,56-59,65-114,127-135,149-153,160-175,181-201,204-207,219-229,231-257,260-291,328-329,337-347,363-367,377-382,392-397
/usr/sausalito/codb/codb.oids reports:
1-51,56-59,65-114,127-135,149-153,160-175,181-201,204-207,219-229,231-257,260-291,328-329,337-347,363-367,377-382,392-397

The lines with the numbers are what we are looking for. They MUST be 
identical. If they are not, then that is the problem.

So if the lines are NOT identical, do this:

Edit /usr/sausalito/codb/codb.oids and remove everything in it. Then copy the 
first numbered line into that file.

Afterwards restart CCEd with this command:

/etc/init.d/cced.init restart

That should fix it.

-- 
With best regards

Michael Stauber