[BlueOnyx:19687] Re: Updated CMU published

[Richard Morgan :: Morgan Web]

Thank you for this Michael.

Just one thing to report is that pigz is causing one of our servers to
almost grind to a halt.

This older, lowish spec (Dell R200, 4Gb) server mostly serves image files so
there is a lot of data which doesn't compress that well. This was previously
fine but yesterday and today there was an outage.  Is there an option
available to preserve the previous behaviour?

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1144 root      16   0 35440 2936  464 S 50.9  0.1   4:45.61 pigz
16667 mysql     15   0  152m  20m 3752 S  2.0  0.6 593:16.48 mysqld
    1 root      15   0  2172  360  332 S  0.0  0.0   8:57.75 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   1:34.59 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0  10:11.54 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.01 watchdog/0

Thanks, Richard


-----Original Message-----
From: Blueonyx [mailto:blueonyx-bounces at mail.blueonyx.it] On Behalf Of
Michael Stauber
Sent: 09 June 2016 00:49
To: BlueOnyx General Mailing List
Subject: [BlueOnyx:19669] Updated CMU published


Hi all,

One of the more complex pieces of code we are maintaining is CMU - our
Cobalt Migration Utility. It had fallen into a slightly deplorable state. It
was still mostly working, but while doing so it was groaning, moaning and
bitching. No more of that!

The following fixes and improvements were made:

Speed:
=======

We all know a CMU-Export (and import) can take its sweet time. Walzing
through all those tarballs? That's slow. Especially if there is lots of
data. The tarballs CMU generates for Vsites and Users are regular TAR
archives packed with Gnuzip. TAR is relatively fast, but the Gnuzip part is
what takes the longest.

For that reason we now switched CMU to use the command line utility 'pigz'
to compress the tarballs. Mind you: This is still in Gnuzip format. So it
can be unpacked with regular Gnuzip and is 1:1 compatible.
It's just that 'pigz' is multiprocessor capable and will use all available
CPU cores for parallel processing of the compressed archive.

The speed increase this provides naturally depends on a lot of factors.
Like how many CPU cores your server has available and also how good (or
bad) your files can be compressed.

But I've seen speed increases between 5-10 times for a regular CMU-Export
during testing. \o/

The CMU-Import now also goes considerably faster than before, as it also
uses 'pigz' when unpacking the tarballs.


Bugfixes:
=========

- The warning message "Subroutine Compress::Zlib::gzFile::gzseek
  redefined" is now gone.

- Helptexts for cmuExport and cmuImport (available via "-h" parameter)
  have been cleaned up and clarified.

- Incorrect helptext for cmuImport parameter "-s" corrected. It is now
  "import server-admins (Resellers)" as it should be.

- Various quota related issues improved and fixed. In the past you
  would sometimes get some weird characters on the screen if a
  cmuImport did run into quota issues. This will no longer happen.

- Various other fixes.


Summary:
========

Both a cmuExport and a cmuImport should now be a much faster and much
cleaner experience than before.

There are still a few identified and open issues, which we will address in
future CMU updates. Such as the current inability to cmuImport subdomains.
Another room for improvement is the direct export/import of DNS and MySQL
via CMU, which will be added in the close future. As is both DNS and MySQL
must be exported/imported separately via the methods explained in our
Migration Guide.

The updated CMU is available as YUM update for all versions of BlueOnyx.

--
With best regards

Michael Stauber
_______________________________________________
Blueonyx mailing list
Blueonyx at mail.blueonyx.it
http://mail.blueonyx.it/mailman/listinfo/blueonyx