[BlueOnyx:19502] Re: 5209R systemd hangs on cced.init

[Michael Stauber]

Hi Chris,

> Not sure if anyone else has run across this.  We have a box that, when
> rebooted, does not come back from the reboot and cannot be remotely
> accessed.  After looking into it I found that systemd had hung at the
> boot process.  Apparently, cced.init is far up the front of the list for
> this, and for whatever reason the server hangs there.

I recall this happening when I tried to find out which boot-order I
could give CCEd early in the 5209R development.

The boot-order of CCEd is defined in the CCEd Unit file for Systemd:

#> cat /usr/lib/systemd/system/cced.init.service
[Unit]



Description=CCEd - The Cobalt Configuration Engine Daemon
After=network.target remote-fs.target nss-lookup.target
Before=admserv.service netfs.service sendmail.service httpd.service
sshd.service
....

So it should start once the network is up, any remote file systems are
mounted and the nss-service is ready. That usually does the trick and
once that was configured it seemed to work fine all the time. Hence I
left it at that.

The question would be what's different on your box? Please check
/usr/lib/systemd/system/cced.init.service and review the entries in the
line starting with "After=" and "Before=". They should be as shown above.

How to debug the start order? This is where it gets really annoying.
With SystemV or Upstart you can easily look at the directory tree and
can figure that out. With Systemd it requires arcane magic and the soul
of your firstborn. :-(

There is also the option to dump an SVG graphics (Really! You can't make
that one up!) with the "systemd-analyze" command. To dump a graphic for
just the cced service you can use this syntax:

systemd-analyze dot 'cced*' | dot -Tsvg > cced.svg

Here is a different syntax to see the "targets" (loosely similar to
runlevels):

systemd-analyze dot --to-pattern='*.target' --from-pattern='*.target' |
dot -Tsvg > targets.svg

My guess? Something shook up the boot order and causes the conflict. It
would be interesting to see what "systemctl list-jobs" reports when the
system hangs during startup. Because we would need to find out which
service it last tried to start before it got stuck, as that service is
probably the culprit.

-- 
With best regards

Michael Stauber