[BlueOnyx:18264] NewLinQ and Shop outage

[Michael Stauber]

Hi all,

We're currently having a bit of an issue (insert British understatement)
with NewLinQ and the shop pages.

Greg is currently working hard to restore services as fast as possible.
But this outage will probably be partially with us the whole day with
services gradually returning one after the other.

Affected services:

Shops:
======

www.compassnetworks.com.au
www.solarspeed.net
shop.blueonyx.it

NewLinQ:
========

newlinq.compassnetworks.com.au
newlinq.blueonyx.it

Wiki:
=====

wiki.blueonyx.it


Here is what happened: It has nothing to do with the SL 6.7 update. That
part is just bad timing. A disk in the RAID array of the master node had
failed. To fix that we set up a scheduled maintenance window to get that
sorted and expected maybe an hour of downtime at the worst. During the
disk replacement (and to ease future replacements) the disks were
switched to hot swappable trays as well.

Sadly things didn't go as planned and the remaining (good) disk in the
array also went bad before it had a chance to finish the sync to the new
replacement drive. \o/

So we're currently doing a full restore from the backups. Due to the
amount of data this will take a couple of hours until everything is
fully restored from last nights snapshots.

Naturally: NewLinQ services will be restored first of all, as its
absence has the biggest impact.

Apologies to all around.

-- 
With best regards

Michael Stauber