[BlueOnyx:24558] Re: Yum update error 5209R - solved

[Michael Stauber]

Hi Darren,

> The repo problem seems to be back again - is there a workaround that we
> can use during outages?
> 
> Could not retrieve mirrorlist
> http://updates.blueonyx.it/index.php?release=7&arch=x86_64 error was
> 
> 14: curl#56 - "Callback aborted"

That should be working again by now.

We had a little hickup with three of our four virtualization nodes
restarting. This took down ~30 VPS's including primary and secondary DNS
and two mirrors. A third mirror and the tertiary DNS is in another
location. And it took down the list, most of the blueonyx.it webpages
and a couple of other odds and sods as well.

After the nodes had rebooted and all VPS's that were supposed to be
running were running. So everything seemed to be fine again.

But that wasn't really the case.

Within the hour all DNS records dropped due to short TTL and the primary
and secondary DNS no longer being connected to the internet. The
tertiary DNS isn't really redundant, so it couldn't take over either.

Why did all VPS's drop off the net despite them running? During a
Container restart on OpenVZ 7 the CT's network disposition is
ARP-broadcasted and it looks like the switch lost these sometime after
the VPS's initially had come back. So any IP packets destined for the
VPS's ended up in nirvana as the last hop wasn't routed to where it
should go. Restarting all VPS's *again* and triggering new ARP seeding
solved that.

I'm in the process of setting up new mirrors anyway (they need a more up
to date base OS than they currently have) and while I do so I'll make
sure that our tertiary DNS can work independently to prevent complete
mirror outages such as this in the future.

-- 
With best regards

Michael Stauber