[BlueOnyx:08260] eth Interface Bug and More!

[Aaron Greenspan]

Hi,

As part of my recent unintentional multi-week experiment getting some 
BlueOnyx servers set up, I ran into an esoteric bug or two.

I've been imaging servers by using RAID 1 to duplicate hard drives 
across machines, starting from one, and then creating another and 
another as necessary. I have four Sun Fire X2200 servers. They all have 
four ethernet ports in the back: two are controlled by an nVidia 
chipset, and two by a Broadcom chipset. They are labeled on the physical 
case as eth0, eth1, eth2 and eth3.

The way the ports get labeled in the operating system, using the same 
terms (eth0 through eth3) is pretty haphazard, meaning that the OS 
labels definitely do not correspond to the physical port labels. 
Suddenly eth3 means eth0. It's very confusing. I doubt there's much that 
can be done about this unless the hardware itself provides metadata 
about which port is which, and I don't know if it does.

At first I thought that this was just a minor annoyance, but it turns 
out that it matters when combined with the RAID 1 imaging method I'd 
been using, because when you take an imaged drive from another CentOS 
server and plug it into a new chassis (with new ethernet adapters that 
each has a new MAC address), what happens is not quite what you'd 
expect: rather than replace the four old MAC addresses with four new MAC 
addresses, udev just concatenates them in 
/etc/udev/rules.d/70-persistent-net.rules. So you end up with eth0 with 
eth7, none of which correspond to anything in reality. This makes it 
really hard to get an internet connection up and running, by the way.

It gets better. When you go to change your IP address in the BlueOnyx 
control panel, you are presented with four sets of fields corresponding 
to your four ethernet adapters. But you can't save. Why can't you save? 
Because there's a JavaScript error. Why is there a JavaScript error? 
Because on line 101 of /usr/sausalito/ui/web/base/network/ethernet.php, 
the PHP variable $interfaces is getting its data from 
$cceClient->findx('Network', array('real' => 1), array(), 'ascii', 
'device');, and that's reporting back that there are in fact eight 
ethernet adapters. There aren't, but as soon as JavaScript hits 
$interfaces[4], it can't find the HTML inputs because they don't exist. 
(This makes it REALLY hard to get an internet connection up and running, 
by the way.)

At the root level the problem is with udev being dumb, but having two 
different ways of determining the number of ethernet adapters (there are 
only four input blocks, which is correct, versus the eight JavaScript 
array elements) does not help.

Another fun fact about using RAID 1 to clone a clean BlueOnyx install 
image: it kills your swap partition. If you ever want that back instead 
of a lame md127 array that comes out of nowhere because the mdadm-style 
(versus the other style, who thought of this insanity and why are there 
two styles?) UUID of the array doesn't match the contents of 
/etc/mdadm.conf, you'll have to do the following:

mdadm --manage --stop /dev/md127
mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
mdadm --misc --detail /dev/md2 (tells you the UUID)
vim /etc/mdadm.conf (paste the UUID in the proper place)
mkswap /dev/md2
swapon /dev/md2

I'm using /dev/md2 here for the array and /dev/sda and /dev/sdb for my 
drives, but fill in your own device handlers as necessary.

Aaron

------------------------------------------------------------------------
Think Computer Corporation <http://www.thinkcomputer.com> 	

*Aaron Greenspan*
President & CEO
Think Computer Corporation

/telephone/ +1 415 670 9350
/toll free/ +1 888 815 8599
/fax/ +1 415 373 3959
/e-mail/ aarong at thinkcomputer.com
/web/ http://www.thinkcomputer.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.blueonyx.it/pipermail/blueonyx/attachments/20110825/cfd98422/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: think.gif
Type: image/gif
Size: 1806 bytes
Desc: not available
URL: <http://mail.blueonyx.it/pipermail/blueonyx/attachments/20110825/cfd98422/attachment.gif>