[BlueOnyx:20211] BlueOnyx Mirror Maintenance

[Michael Stauber]

Hi all,

As you might (hopefully not) have noticed: The BlueOnyx mirrors were
having an outage of several hours today. That should be fixed by now.

Background:

The last few weeks I noticed that the occasional timeouts of the mirrors
incrementally grew, resulting in messages such as this:

Could not retrieve mirrorlist
http://updates.blueonyx.it/index.php?release=6.8&arch=x86_64 error was
12: Timeout on http://updates.blueonyx.it/index.php?release=6.8&arch=x86_64

At first I suspected problems with Varnish and Nginx on the two VPS's
that run updates.blueonyx.it. So I set up monitoring scripts on each
that checked if the local Varnish, Nginx and PHP-FPM were running and
responding without failure message. If they didn't respond a restart of
the three services would be initiated and the problem would be logged.

That resulted in zero logged problems that I hadn't caused myself for
testing purposes. :p

So today I tackled the real problem: The separate VPS with the SQL
backend of the mirrors. The database had grown to a point where it got a
little unwieldy and the 32-bit MySQL server on it wasn't really tweaked
in the most ideal fashion anyway. Under more than average load it didn't
respond in time to queries. YUM has a pretty short timeout and if the
Varnish + Nginx + SQL combo doesn't respond within a few seconds, YUM
throws a fit.

I migrated the SQL VPS to 5209R and MariaDB and tweaked the settings to
coax out more performance. I expected a 5-15 minute downtime, but sadly
things didn't go as planned.

This is now all sorted and the mirrors should work a little bit more
reliably than before. There are still several small fixes and tweaks
that I'll need to apply in the days to come, but these shouldn't cause
as much grief as todays fixes.

-- 
With best regards

Michael Stauber