[BlueOnyx:06215] Re: HP DL380 G3's and HPASM and HPASMCLI

[Darrell D. Mobley]

> From: blueonyx-bounces at blueonyx.it [mailto:blueonyx-bounces at blueonyx.it] 
> On Behalf Of Darrell D. Mobley
> Sent: Saturday, October 16, 2010 9:17 PM
> To: 'BlueOnyx General Mailing List'
> Subject: [BlueOnyx:05592] HP DL380 G3's and HPASM and HPASMCLI
> 
> It seems like every time I change /boot/grub/grub.conf to use 
> the latest kernel released by CentOS, shortly thereafter my server 
> will hang up.  It will respond to pings to the IP address, I can 
> FTP to the server IP and it says connected but never goes any further, 
> DNS doesn't seem to be responding.
> 
> The last time I edit grub.conf was on October 11th at 6:54PM, and it 
> hung up October 12th from 3AM until I found I non-responsive the next 
> morning at 9AM.  I cycled the IP-based power strip, and while that 
> brought it back up, it corrupted a few files, so not enjoyable.
> 
> My server has HPASM running on it along with HPASMCLI, the command 
> line version.  One of the features is a watchdog timer:
> 
> ASR - Automatic Server Recovery.
> 
> hpasmcli> show asr
> ASR timeout is 10 minutes.
> ASR is currently enabled.
> 
> Why is the system sitting for 6 hours without rebooting?

Two times this week I have woke up to find a locked up server.  Each time,
it locks up at 3AM.  It will respond to pings, but will not connect via
HTTP, POP, SMPT, SSH or any other method of connection.  Like I mentioned
above, I have HP's Automatic System Recovery enabled, but it never reboots
because apparently, the server doesn't know it's locked up.  All services
appear to stop, no cron jobs run, I can't figure it out.

I have scoured every log on the site, and here are the last entries recorded
before it locks up:

messages:
Jan  1 03:02:01 web1 cced(smd)[1310]: client [0:1307] has admin rights
Jan  1 03:02:01 web1 cced(smd)[1315]: client [0:1312] has admin rights

cron:
Jan  1 03:02:01 web1 crond[1174]: (root) CMD (/usr/local/sbin/dfix.sh)

audit:
type=USER_ACCT msg=audit(01/01/2011 03:02:01.374:22627) : user pid=1173
uid=root auid=unset subj=kernel msg='PAM: accounting acct=root :
exe=/usr/sbin/crond (hostname=?, addr=?, terminal=cron res=success)' 
type=CRED_ACQ msg=audit(01/01/2011 03:02:01.374:22628) : user pid=1173
uid=root auid=unset subj=kernel msg='PAM: setcred acct=root :
exe=/usr/sbin/crond (hostname=?, addr=?, terminal=cron res=success)' 
type=LOGIN msg=audit(01/01/2011 03:02:01.375:22629) : login pid=1173
uid=root old auid=unset new auid=root old ses=4294967295 new ses=3632 
type=USER_START msg=audit(01/01/2011 03:02:01.382:22630) : user pid=1173
uid=root auid=root subj=kernel msg='PAM: session open acct=root :
exe=/usr/sbin/crond (hostname=?, addr=?, terminal=cron res=success)' 
type=CRED_DISP msg=audit(01/01/2011 03:02:01.916:22631) : user pid=1173
uid=root auid=root subj=kernel msg='PAM: setcred acct=root :
exe=/usr/sbin/crond (hostname=?, addr=?, terminal=cron res=success)' 
type=USER_END msg=audit(01/01/2011 03:02:01.917:22632) : user pid=1173
uid=root auid=root subj=kernel msg='PAM: session close acct=root :
exe=/usr/sbin/crond (hostname=?, addr=?, terminal=cron res=success)' 

maillog:
Jan  1 03:02:44 web1 dovecot: pop3-login: Login: user=<jimamos>,
method=PLAIN, rip=67.228.24.231, lip=208.77.219.104
Jan  1 03:02:44 web1 dovecot: POP3(jimamos): Disconnected: Logged out
top=0/0, retr=0/0, del=0/0, size=0
Jan  1 03:02:52 web1 dovecot: pop3-login: Login: user=<site12>,
method=PLAIN, rip=216.17.205.247, lip=208.77.219.98
Jan  1 03:02:53 web1 dovecot: POP3(site12): Disconnected: Logged out
top=0/0, retr=0/0, del=0/0, size=0

access_log:
www.domainname.com 66.69.60.93 - - [01/Jan/2011:03:02:57 -0500] "GET
/banners/adlog.php?bannerid=90&clientid=48&zoneid=0&source=&block=0&capping=
0&cb=688b91f1455489eb86bf4b0df9135216 HTTP/1.1" 200 43
"http://www.domainname.com/content/view/3071/38/1/5/" "Mozilla/5.0
(compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"