[BlueOnyx:12014] Re: Varnish Cache

[Michael Stauber]

Hi Frank,

> What I don't understand, that's if it is complicated to consider this
> tool in Blueonyx. In fact, in its simpliest usage (caching sites), the 
> only thing to do is to put Apache on another port, and Varnish listening 
> to port 80.
> Is there terribly difficult to enhanced the GUI to permit default port 
> modification from 80 to, e.g. 8080 ? In a first time this may be done 
> perhaps even in a conf file, not GUI.
> 
> Does it involves only to put "8080" instead of "80" in the siteX files, 
> or very much more ?

I wish it was that simple.

Yes, the GUI side of things to put Apache on another port aren't that
terribly complicated to do. It gets complicated in a scenario where you
have more than one Vsite on your BlueOnyx. Which is typically the case.

Say 20 Vsites run on one IP. You want to switch one Vsite to
127.0.0.1:80 instead, so that you can put a Varnish in front of it.

Well, you can't.

You'd have to switch all other 19 Vsites as well, or you'd have to
assign a different IP to them. You also need to instruct Apache not to
bind itself to all available IP's, which is done somewhere else
(directly in httpd.conf).

Now if you switch all 20 Vsites to 127.0.0.1:80, you could then put
Varnish in front of them.

But fun and games: Your Varnish configuration now needs rewrite rules to
handle all 20 Vsites - including *all* their web server aliases. Because
if Varnish listens to the incoming port 80 traffic on that single
external IP, it needs to set the "req.http.host" variable to the FQDN
that was requested, or the underlying webserver wouldn't know which of
the 20 Vsites on that IP it ought to serve.

It may get worse: One of the 20 sites might have SSL enabled. So we'd
also bind Apache's port 443 to localhost and need Varnish rules to deal
with port 443 traffic.

Now it gets worse: The client of site #15 needs a special Varnish rule
as he doesn't want to cache images out of his /news/ directory. But the
client of site #2 also has a /news/ directory that he wants cached
totally. Which is possible, but then we'd need separate Varnish rules
for for those two Vhosts (and perhaps the rest of them as well).

Eventually, after weeks of coding we might end up with a more or less
useful Varnish Cache implementation in BlueOnyx that's well enough
integrated into the GUI. Only to have you realize that the client with
that particular demand would have been better off using a dedicated
server (or VPS) of his own. Because without a lot of tinkering none of
his PHP scripts was cached by Varnish, because all the scripts used
session_start() in every request, which made caching impossible. What
the client might have needed was a lot of ESI coding in his site
template to make the pages more cache-friendly (and a highly customized
Varnish config), but he didn't know enough about it to ponder these
possibilities.

Even if we finally might have a GUI to tweak all the Varnish related
options: Whenever someone applies the changes, the cache needs to be
restarted, which will affect all cached Vsites.

All in all I'd say: The Varnish cache *might* be useful in a shared
hosting environment, but that usefulness will be quite limited. It's not
a "one size fits all" solution. The usefulness depends on the served
content for which you might need a specifically tweaked configuration.
This required configuration adds a layer of complexity that will make
troubleshooting and/or servicing by non Linux experts even more of a
challenge.

My bottom line: It's possible and it is considered for the re-themed
next version BlueOnyx. However, if a client requests a Varnish cache,
then the likeliness that he needs a dedicated server to best cater his
very individual needs is very high. Or he might not know what he's
asking for.

Related to this topic: The BlueOnyx team is using Varnish cache for the
site "updates.blueonyx.it", which is the main YUM repository. This move
was necessary for adding a bit more resiliency and robustness. We do
cache a few things such as the XML files, but mostly the Varnish cache
acts as frontend to round-robin the HTTP requests to those HTTP servers
that have replied positively to periodically run availability checks. We
also use Varnish to filter requests. People who accidentally have set
"updates.blueonyx.it" as SWupdate server are redirected to
"newlinq.blueonyx.it", the highly intrusive robot-crawler of the search
engine Baidu is sent off to index their own domain and we also do a few
other funky things that slow down or divert unwanted traffic.

At the same time it's used as a bit of a learning experience to get more
up to date on all the useful things that Varnish can and can't do.

-- 
With best regards

Michael Stauber