[BlueOnyx:19691] Re: iptables not starting /fresh 5209R install

[Michael Stauber]

Hi Steve,

> Are you sure? I have a box that has a mash-up of the default redhat,
> with all the acctout acctin stuff bolted on.

I checked and we're in a somewhat inconsistent state there. The hourly
cronjob log_traffic got updated a while ago (18th Dec 2015) and *that*
version no longer creates an updated /etc/sysconfig/iptables as we no
longer deemed that to be necessary.

That assumption was wrong and bit us in the ass.

The RPM 'iptables-services' brings its own /etc/sysconfig/iptables with
default rules aboard. But it has 'config-noreplace' set for that
particular file.

So if there is already a /etc/sysconfig/iptables present (like from an
old base-sitestats RPM with the old log_traffic cronjob!), then it will
*not* overwrite the good rules with the bad default ones.

However: Any BlueOnyx 5209R installed with the 5209R ISO from 14th
January 2016 (and later) would start with the new log_traffic cronjob
that would never create /etc/sysconfig/iptables in first place. In that
case the install of the RPM 'iptables-services' would deploy the 'bad'
default firewall rules.

I did test the update on two boxes (a VM installed of an ISO and an
OpenVZ VPS). But both were older and already had an
/etc/sysconfig/iptables in place. So these boxes were safe and the
update caused no issues.

The only good news is that it didn't affect each and any 5209R install.
Due to the timing between deploying the update and rolling it back it
was out for ten hours (9. Jun 16:39 to 10. Jun 02:33 GMT+5). And even
then it affected only builds from CD's newer than 14th January 2016.

Still: This was pretty nightmarish and I'm really sorry it happened.

My sincere apologies to all who were affected!


Permanent fixes to prevent this from happening again:
======================================================

I'm now in the process of building and testing a
'base-sitestats-scripts' RPM that will do the following:

1.) It still installs 'iptables-services' as we need it for the Systemd
Unit-File of iptables that it contains.

2.) /etc/cron.hourly/log_traffic now makes sure that there is a safe
/etc/sysconfig/iptables file present. That will prevent that future OS
updates will bring the default (bad!) firewall rules back.

3.) The post-install script of our new 'base-sitestats-scripts' has a
rather lengthy mechanism in it. It could be shorter, but for clarity and
ease of understanding I left it longer:

-----------------------------------------------------------
%post

if [ -f /bin/systemctl ]; then
  ### Disable Firewalld and make sure it's dead:
  # Stop and disable firewalld:
  systemctl stop firewalld.service
  systemctl disable firewalld.service
fi

# Turn module unload off for iptables:
/bin/sed -i -e
's@^IPTABLES_MODULES_UNLOAD="yes"@IPTABLES_MODULES_UNLOAD="no"@'
/etc/sysconfig/iptables-config

# Check if APF is present:
if [ -d /etc/apf ];then

  # APF present. Disable and stop iptables:
  rm -f /etc/sysconfig/iptables
  touch /etc/sysconfig/iptables
  echo "# Empty, because APF is present" > /etc/sysconfig/iptables
  systemctl disable iptables.service
  systemctl stop iptables.service &>/dev/null || :

else

  # Flush existing iptables rules:
  iptables --flush

  if [$1 -eq 1]; then

    # New Install

    # Zap existing /etc/sysconfig/iptables
    rm -f /etc/sysconfig/iptables
    touch /etc/sysconfig/iptables
    echo "# Empty, because log_traffic hasn't run yet." >
/etc/sysconfig/iptables

    # Enable iptables:
    systemctl enable iptables.service

  else

    # Upgrade of already installed RPM:

    # Zap existing /etc/sysconfig/iptables
    rm -f /etc/sysconfig/iptables
    touch /etc/sysconfig/iptables
    echo "# Empty, because log_traffic hasn't run yet." >
/etc/sysconfig/iptables

    # Generate accounting rules for iptables:
    /etc/cron.hourly/log_traffic &>/dev/null || :

    # Save new iptables rules:
    iptables-save > /etc/sysconfig/iptables

    # Enable iptables:
    systemctl enable iptables.service

    # Start iptables:
    if [ $1 -gt 1 ]; then
      # RPM upgrade
      systemctl restart iptables.service &>/dev/null || :
    fi

  fi
fi

-----------------------------------------------------------

There are provisions for APF in this, because APF disables the service
"iptables" and replaces it. We still want to make darn sure that there
is a safe /etc/sysconfig/iptables present - even if APF won't need it.

If APF is present:
===================

- A safe /etc/sysconfig/iptables with just a comment line is written.
- Service iptables gets stopped and disabled.

If it's a new Install (off the ISO for example):
=================================================

- Firewall rules get flushed.
- A safe /etc/sysconfig/iptables with just a comment line is written
- Service iptables gets enabled.

A restart of iptables is not needed there, as this covers first installs
off the CD before the system is booted for real for the first time.

Install as YUM update:
======================

- Firewall rules get flushed.
- A safe /etc/sysconfig/iptables with just a comment line is written
- Cronjob log_traffic is run. That will replace /etc/sysconfig/iptables
with a "good" firewall config that only has accounting rules in it. -
- Service iptables gets enabled and restarted

That should take care of all eventualities. I'm currently in the process
of testing this under all the plausible scenarios that I can imagine.

-- 
With best regards

Michael Stauber