[BlueOnyx:18362] Re: 5209R: Another app is currently holding the xtables lock

[Michael Stauber]

Hi Chris,

> We have a customer on a 5209R system that has, since September 11, been 
> receiving cron messages stating "Another app is currently holding the 
> xtables lock; waiting for it to exit..."
> 
> This particular server is also using the APF Firewall plugin.
> 
> Any ideas why this may be happening, and what might be done to stop it?

Yeah, this also annoys me to no end. :-(

It only happens on 5209R and CentOS7, though. But it's bad enough as it
generates quite a number of unnecessary emails.

The Active Monitor component of APF /usr/sausalito/swatch/bin/am_apf.pl
checks if a certain iptables chain is present and if it is, then we can
assume that the firewall rules are in place.

On all platforms (but 5209R) it uses this:

$status = `iptables -L -n|grep "^Chain TDENY"|wc -l 2>&1`;

Which works just fine. But iptables on CentOS7 is a bit newer and has
locking mechanisms. If something is currently modifying or reloading the
chains, then another access to iptables (even if it is read only) will
throw a locking error. So they also added the "-w" switch to iptables on
CentOS7. That's supposed to not let the pending request fail in case of
a lock, but to make it wait until the lock is resolved.

>From the iptables manpage:

----------------------------------------------------------
 -w, --wait
 Wait  for  the xtables lock. To prevent multiple instances of the
program from running concurrently, an attempt will be made to obtain an
exclusive lock at launch. By default, the program will exit if the lock
cannot be obtained. This option will make the program wait until the
exclusive lock can be obtained.
----------------------------------------------------------

So for 5209R the Active Monitor component uses this instead:

$status = `iptables -w -L -n|grep "^Chain TDENY"|wc -l 2>&1`;

Which is the same, but with the addition of the "-w" switch. Which
doesn't exist on EL6 or EL5.

The sad part: It still buggers out eventually if an exclusive lock
cannot be obtained and then it throws the "Another app is currently
holding the xtables lock; waiting for it to exit..." message.

I'm royally displeased with this behavior and am exploring other options
at the moment how we can get it to work properly on CentOS7 without
these spurious messages. Once I have it figured out I'll publish an update.

-- 
With best regards

Michael Stauber