[BlueOnyx:21780] Re: Memcached security issue

[Michael Stauber]

Hi Richard,

> Secondly, has anyone experienced this and is a server-wide fix or extra
> limitations Memcached available? What is Memcached used for on BX and it
> is it possible to remove it?

During our "need for speed" initiative for BlueOnyx we looked at certain
ways to improve the speed of CCEd to accelerate the GUI interface. That
development drive has over the years taken various steps that produced
considerable results. However, not all of them were fruitful.

In December 2015 we had published a version of CCEd that used
"memcached" to cache certain frequent CCEd transactions.

Initial tests were promising enough to warrant a release as regular YUM
update, but during the course of the next few days users reported
problems related to multi-user/multi-language caching issues that we had
not foreseen and which put the whole approach of using "memcached" into
question.

So early in January 2016 I published a rollback that stopped and
disabled memcached and reinstalled a CCEd that no longer depended on
"memcached".

Hence the service "memcached" (if installed at all) should be inactive
on a BlueOnyx by default. There is no legitimate process or service on a
fully updated BlueOnyx that needs it and the RPM can in fact be
deinstalled as well:

rpm -e memcached
... or ...
yum remove memcached

I have no idea why it was still enabled on your server and of course it
is highly regrettable that the service was abused.

Even then it raises the question how *that* could have happened, as
"memcached" by default only binds to 127.0.0.1:11211 and is therefore
only accessible by local services and processes on your server and
cannot be reached remotely:

[root at 5208r ~]# /sbin/service memcached restart
Stopping memcached:                                        [FAILED]
Starting memcached:                                        [  OK  ]

[root at 5208r ~]# netstat -tupan|grep LISTEN|grep memcached
tcp        0      0 127.0.0.1:11211             0.0.0.0:*
   LISTEN      7483/memcached

So in order to exploit "memcached" and to use it to run attacks against
other servers someone would have had to gain elevated access on your
BlueOnyx in first place, because "memcached" could not be manipulated
and tricked into running attacks from the outside as it simply wasn't
reachable from the outside on its own.

This needs further investigation and I'd be glad to lend a hand to see
what was going in. Please use the GUI to submit a "Support Request"
(under "Software Updates") and tick the checkbox "Allow access" to
submit temporary login details to your server in a secure fashion.

Beyond that: As we don't need "memcached" on a BlueOnyx anymore, but
some servers still might have it installed I will publish an immediate
YUM update that will "obsolete" the RPM "memcached" to make sure that
it's removed at once. While this should not be necessary due to the
service being configured for localhost usage only *and* the service
being disabled by default I'd rather like to play it safe here.

-- 
With best regards

Michael Stauber