[BlueOnyx:07103] Re: web1.uhostme.com Yum Update output for 04-10-11

Tue Apr 19 08:35:52 -05 2011

I spoke too soon.  My experiment failed.  It just waited another 24 hours to fail.

After I edited grub.conf to change default=1 to default=0 to utilize the new kernel, rebooted the machines and then power-cycled the power strip to force a cold reboot, the machine made it through the first night alright only to stop listening to HTTP, FTP, SSH, POP, etc. last night. Only PING worked.  Because the machine was still running but the not listening, the HP dead man timer never kicks in.  Why this machine only does this when a new kernel comes in via YUM and grub.conf is edited to change “default=1” to “default=2” and rebooted, I have NO IDEA.  I am tired of trying to figure it out, the system can run the old kernel in the default=1 slot until the end of time for all I care.  

From: blueonyx-bounces at mail.blueonyx.it [mailto:blueonyx-bounces at mail.blueonyx.it] On Behalf Of Darrell D. Mobley
Sent: Monday, April 18, 2011 12:14 PM
To: 'BlueOnyx General Mailing List'
Subject: [BlueOnyx:07090] Re: web1.uhostme.com Yum Update output for 04-10-11

With the recent kernel update, I tried another experiment:

I rebooted to load the new kernel, then stopped as many services as possible and then hit the IP-based power switch.  No middle of the night lockup.  This experiment worked.

So this lets me know that I have to do a cold reboot after a new kernel comes in and I update grub.conf to set default=0.  Since power cycling the server isn’t good for the RAID array, all I need now is to get my colo facility to attach a network cable to the iLO port.

From: blueonyx-bounces at mail.blueonyx.it [mailto:blueonyx-bounces at mail.blueonyx.it] On Behalf Of Darrell D. Mobley
Sent: Friday, April 15, 2011 9:14 AM
To: 'BlueOnyx General Mailing List'
Subject: [BlueOnyx:07049] Re: web1.uhostme.com Yum Update output for 04-10-11

Well, the experiment failed.  It seems no matter how many times I reboot the server after a new kernel comes in via YUM and I edit grub.conf, changing “default=1” to “default=0”, it hangs in the middle of the night, stops listening on HTTP, FTP, SSH, etc. With monitoring enabled, my colo host saw it stall at 3:33AM and power cycled it.  It sure would be nice to figure out WHY this is happening because I had power cycling that six drive RAID5.  The only clues I can come up with now are:

Apr 10 06:06:23 web1 init: Trying to re-exec init

Apr 10 06:06:32 web1 init: Trying to re-exec init

Apr 15 03:12:59 web1 init: Trying to re-exec init

The 10th is when the big YUM choke came through, bearing a new kernel.  And of course, last night I edited grub.conf to change default=1 to default=0.

Of course every 30 minutes, it’s trying to clear out those 44 failed logins and failing, because they keep reappearing in the GUI.

From: blueonyx-bounces at mail.blueonyx.it [mailto:blueonyx-bounces at mail.blueonyx.it] On Behalf Of Darrell D. Mobley
Sent: Thursday, April 14, 2011 9:20 PM
To: 'BlueOnyx General Mailing List'
Subject: [BlueOnyx:07040] Re: web1.uhostme.com Yum Update output for 04-10-11

I had not rebooted, but I have now.  Twice. J

Why you might ask?

It’s an experiment.  My HP DL380 G3 likes to hang up at 4AM on the morning after a new kernel comes in via YUM and I edit grub.conf, changing “default=1” to “default=0”. After it hangs up, if I reboot it, it runs fine until the next time a new kernel comes in via YUM and I edit grub.conf again to change the default kernel to the newest one.  I don’t know why.  (By hang up, I mean it stops answering HTTP, FTP, SSH, etc., but responds to PINGs fine.  So it’s not locked up, it just quits responding.)  

So my experiment tonight is to change the default kernel to the latest “0”, reboot it, and then reboot it again.  I want to see if the second reboot up front will work the same magic it has in the past, just  without it sitting 5 hours waiting on me to get up and cycle the power strip.

From: blueonyx-bounces at mail.blueonyx.it [mailto:blueonyx-bounces at mail.blueonyx.it] On Behalf Of Michael Stauber
Sent: Thursday, April 14, 2011 5:44 PM
To: BlueOnyx General Mailing List
Subject: [BlueOnyx:07037] Re: web1.uhostme.com Yum Update output for 04-10-11

Hi Darrell,

> I noticed tmpwatch was updated, now there are both tmpwatch files in the

> cron.daily folder:

> 

> warning: /etc/cron.daily/tmpwatch created as

> /etc/cron.daily/tmpwatch.rpmnew

You can remove /etc/cron.daily/tmpwatch.rpmnew

> Ever since the big YUM choke came through on the 10th, none of root's cron

> jobs were running.  Crond was running, anyone know what happened and why?

Did you reboot since installing all updates? If not, please do. A lot of processes don't like it of glibc or other libraries are replaced while the services are still running.

-- 

With best regards

Michael Stauber

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.blueonyx.it/pipermail/blueonyx/attachments/20110419/f4f39b9e/attachment.html>