<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Verdana","sans-serif";
color:#1F497D;
font-weight:normal;
font-style:normal;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Verdana","sans-serif";
color:#1F497D;
font-weight:normal;
font-style:normal;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Verdana","sans-serif";
color:#1F497D;
font-weight:normal;
font-style:normal;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Verdana","sans-serif";
color:#1F497D;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>I spoke too soon. My experiment failed. It just waited another 24 hours to fail.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>After I edited grub.conf to change default=1 to default=0 to utilize the new kernel, rebooted the machines and then power-cycled the power strip to force a cold reboot, the machine made it through the first night alright only to stop listening to HTTP, FTP, SSH, POP, etc. last night. Only PING worked. Because the machine was still running but the not listening, the HP dead man timer never kicks in. Why this machine only does this when a new kernel comes in via YUM and grub.conf is edited to change “default=1” to “default=2” and rebooted, I have NO IDEA. I am tired of trying to figure it out, the system can run the old kernel in the default=1 slot until the end of time for all I care. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> blueonyx-bounces@mail.blueonyx.it [mailto:blueonyx-bounces@mail.blueonyx.it] <b>On Behalf Of </b>Darrell D. Mobley<br><b>Sent:</b> Monday, April 18, 2011 12:14 PM<br><b>To:</b> 'BlueOnyx General Mailing List'<br><b>Subject:</b> [BlueOnyx:07090] Re: web1.uhostme.com Yum Update output for 04-10-11<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>With the recent kernel update, I tried another experiment:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>I rebooted to load the new kernel, then stopped as many services as possible and then hit the IP-based power switch. No middle of the night lockup. This experiment worked.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>So this lets me know that I have to do a cold reboot after a new kernel comes in and I update grub.conf to set default=0. Since power cycling the server isn’t good for the RAID array, all I need now is to get my colo facility to attach a network cable to the iLO port.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> blueonyx-bounces@mail.blueonyx.it [mailto:blueonyx-bounces@mail.blueonyx.it] <b>On Behalf Of </b>Darrell D. Mobley<br><b>Sent:</b> Friday, April 15, 2011 9:14 AM<br><b>To:</b> 'BlueOnyx General Mailing List'<br><b>Subject:</b> [BlueOnyx:07049] Re: web1.uhostme.com Yum Update output for 04-10-11<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Well, the experiment failed. It seems no matter how many times I reboot the server after a new kernel comes in via YUM and I edit grub.conf, changing “default=1” to “default=0”, it hangs in the middle of the night, stops listening on HTTP, FTP, SSH, etc. With monitoring enabled, my colo host saw it stall at 3:33AM and power cycled it. It sure would be nice to figure out WHY this is happening because I had power cycling that six drive RAID5. The only clues I can come up with now are:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Apr 10 06:06:23 web1 init: Trying to re-exec init<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Apr 10 06:06:32 web1 init: Trying to re-exec init<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Apr 15 03:12:59 web1 init: Trying to re-exec init<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>The 10<sup>th</sup> is when the big YUM choke came through, bearing a new kernel. And of course, last night I edited grub.conf to change default=1 to default=0.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Of course every 30 minutes, it’s trying to clear out those 44 failed logins and failing, because they keep reappearing in the GUI.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> blueonyx-bounces@mail.blueonyx.it [mailto:blueonyx-bounces@mail.blueonyx.it] <b>On Behalf Of </b>Darrell D. Mobley<br><b>Sent:</b> Thursday, April 14, 2011 9:20 PM<br><b>To:</b> 'BlueOnyx General Mailing List'<br><b>Subject:</b> [BlueOnyx:07040] Re: web1.uhostme.com Yum Update output for 04-10-11<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>I had not rebooted, but I have now. Twice. </span><span style='font-size:10.0pt;font-family:Wingdings;color:#1F497D'>J</span><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Why you might ask?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>It’s an experiment. My HP DL380 G3 likes to hang up at 4AM on the morning after a new kernel comes in via YUM and I edit grub.conf, changing “default=1” to “default=0”. After it hangs up, if I reboot it, it runs fine until the next time a new kernel comes in via YUM and I edit grub.conf again to change the default kernel to the newest one. I don’t know why. (By hang up, I mean it stops answering HTTP, FTP, SSH, etc., but responds to PINGs fine. So it’s not locked up, it just quits responding.) <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>So my experiment tonight is to change the default kernel to the latest “0”, reboot it, and then reboot it again. I want to see if the second reboot up front will work the same magic it has in the past, just without it sitting 5 hours waiting on me to get up and cycle the power strip.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> blueonyx-bounces@mail.blueonyx.it [mailto:blueonyx-bounces@mail.blueonyx.it] <b>On Behalf Of </b>Michael Stauber<br><b>Sent:</b> Thursday, April 14, 2011 5:44 PM<br><b>To:</b> BlueOnyx General Mailing List<br><b>Subject:</b> [BlueOnyx:07037] Re: web1.uhostme.com Yum Update output for 04-10-11<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p style='margin:0in;margin-bottom:.0001pt'><span style='font-family:"Arial","sans-serif"'>Hi Darrell,<o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>> I noticed tmpwatch was updated, now there are both tmpwatch files in the<o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>> cron.daily folder:<o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>> <o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>> warning: /etc/cron.daily/tmpwatch created as<o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>> /etc/cron.daily/tmpwatch.rpmnew<o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>You can remove /etc/cron.daily/tmpwatch.rpmnew<o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>> Ever since the big YUM choke came through on the 10th, none of root's cron<o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>> jobs were running. Crond was running, anyone know what happened and why?<o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>Did you reboot since installing all updates? If not, please do. A lot of processes don't like it of glibc or other libraries are replaced while the services are still running.<o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>-- <o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>With best regards<o:p></o:p></span></p><p style='margin:0in;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0;-qt-user-state:0'><span style='font-family:"Arial","sans-serif"'>Michael Stauber<o:p></o:p></span></p></div></div></div></div></div></body></html>