[BlueOnyx:22523] Fail2ban issues on 5209r

Richard Owen richard at pelicanit.co.uk
Sun Nov 25 04:32:30 -05 2018 

Hi

We have had continuous  fail2ban issues ever since we moved to the new 5209r system from 5208.
We have had other issues on the 5209r  which meant we then migrated again last night, to new 5209r
On a new Aventurin{e} node . and are monitoring to see if things have improved in regards stability.

However, We still have issues with fail2ban on this new 5209r 

If enabled ( we had to disable it on previous 5209r due to the same issue)  we get the following errors, and then around 03:00 fail2ban takes 100% CPU and prevents other cron jobs from running,  for example, backups, we then have to kill fail2ban, this releases the CPU, and the other cron jobs then start.

Fail2 ban errors below
2018-11-25 09:17:39,873 fail2ban.server         [13806]: INFO    Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.6
2018-11-25 09:17:39,924 fail2ban.database       [13806]: INFO    Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2018-11-25 09:17:39,933 fail2ban.jail           [13806]: INFO    Creating new jail 'sshd'
2018-11-25 09:17:40,047 fail2ban.jail           [13806]: INFO    Jail 'sshd' uses systemd {}
2018-11-25 09:17:40,086 fail2ban.jail           [13806]: INFO    Initiated 'systemd' backend
2018-11-25 09:17:40,089 fail2ban.filter         [13806]: INFO    Set maxRetry = 2
2018-11-25 09:17:40,090 fail2ban.filter         [13806]: INFO    Set jail log file encoding to ISO-8859-1
2018-11-25 09:17:40,091 fail2ban.actions        [13806]: INFO    Set banTime = 3600
2018-11-25 09:17:40,092 fail2ban.filter         [13806]: INFO    Set findtime = 1600
2018-11-25 09:17:40,092 fail2ban.filter         [13806]: INFO    Set maxlines = 10
2018-11-25 09:17:40,213 fail2ban.filtersystemd  [13806]: INFO    Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd'
2018-11-25 09:17:40,219 fail2ban.jail           [13806]: INFO    Creating new jail 'sshd-ddos'
2018-11-25 09:17:40,220 fail2ban.jail           [13806]: INFO    Jail 'sshd-ddos' uses systemd {}
2018-11-25 09:17:40,231 fail2ban.jail           [13806]: INFO    Initiated 'systemd' backend
2018-11-25 09:17:40,233 fail2ban.filter         [13806]: INFO    Set maxRetry = 2
2018-11-25 09:17:40,235 fail2ban.filter         [13806]: INFO    Set jail log file encoding to ISO-8859-1
2018-11-25 09:17:40,235 fail2ban.actions        [13806]: INFO    Set banTime = 3600
2018-11-25 09:17:40,236 fail2ban.filter         [13806]: INFO    Set findtime = 1600
2018-11-25 09:17:40,242 fail2ban.filtersystemd  [13806]: INFO    Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd'
2018-11-25 09:17:40,248 fail2ban.jail           [13806]: INFO    Creating new jail 'proftpd'
2018-11-25 09:17:40,249 fail2ban.jail           [13806]: INFO    Jail 'proftpd' uses systemd {}
2018-11-25 09:17:40,258 fail2ban.jail           [13806]: INFO    Initiated 'systemd' backend
2018-11-25 09:17:40,261 fail2ban.filter         [13806]: INFO    Set maxRetry = 2
2018-11-25 09:17:40,262 fail2ban.filter         [13806]: INFO    Set jail log file encoding to ISO-8859-1
2018-11-25 09:17:40,263 fail2ban.actions        [13806]: INFO    Set banTime = 3600
2018-11-25 09:17:40,264 fail2ban.filter         [13806]: INFO    Set findtime = 1600
2018-11-25 09:17:40,290 fail2ban.jail           [13806]: INFO    Creating new jail 'sendmail-auth'
2018-11-25 09:17:40,290 fail2ban.jail           [13806]: INFO    Jail 'sendmail-auth' uses systemd {}
2018-11-25 09:17:40,297 fail2ban.jail           [13806]: INFO    Initiated 'systemd' backend
2018-11-25 09:17:40,299 fail2ban.filter         [13806]: INFO    Set maxRetry = 2
2018-11-25 09:17:40,301 fail2ban.filter         [13806]: INFO    Set jail log file encoding to ISO-8859-1
2018-11-25 09:17:40,301 fail2ban.actions        [13806]: INFO    Set banTime = 3600
2018-11-25 09:17:40,302 fail2ban.filter         [13806]: INFO    Set findtime = 1600
2018-11-25 09:17:40,308 fail2ban.filtersystemd  [13806]: INFO    Added journal match for: '_SYSTEMD_UNIT=sendmail.service'
2018-11-25 09:17:40,313 fail2ban.jail           [13806]: INFO    Creating new jail 'sendmail-reject'
2018-11-25 09:17:40,314 fail2ban.jail           [13806]: INFO    Jail 'sendmail-reject' uses systemd {}
2018-11-25 09:17:40,317 fail2ban.jail           [13806]: INFO    Initiated 'systemd' backend
2018-11-25 09:17:40,319 fail2ban.filter         [13806]: INFO    Set maxRetry = 2
2018-11-25 09:17:40,320 fail2ban.filter         [13806]: INFO    Set jail log file encoding to ISO-8859-1
2018-11-25 09:17:40,321 fail2ban.actions        [13806]: INFO    Set banTime = 3600
2018-11-25 09:17:40,322 fail2ban.filter         [13806]: INFO    Set findtime = 1600
2018-11-25 09:17:40,322 fail2ban.filter         [13806]: INFO    Set maxlines = 10
2018-11-25 09:17:40,355 fail2ban.filtersystemd  [13806]: INFO    Added journal match for: '_SYSTEMD_UNIT=sendmail.service'
2018-11-25 09:17:40,361 fail2ban.jail           [13806]: INFO    Creating new jail 'dovecot'
2018-11-25 09:17:40,361 fail2ban.jail           [13806]: INFO    Jail 'dovecot' uses systemd {}
2018-11-25 09:17:40,365 fail2ban.jail           [13806]: INFO    Initiated 'systemd' backend
2018-11-25 09:17:40,366 fail2ban.filter         [13806]: INFO    Set maxRetry = 2
2018-11-25 09:17:40,368 fail2ban.filter         [13806]: INFO    Set jail log file encoding to ISO-8859-1
2018-11-25 09:17:40,368 fail2ban.actions        [13806]: INFO    Set banTime = 3600
2018-11-25 09:17:40,369 fail2ban.filter         [13806]: INFO    Set findtime = 1600
2018-11-25 09:17:40,404 fail2ban.filtersystemd  [13806]: INFO    Added journal match for: '_SYSTEMD_UNIT=dovecot.service'
2018-11-25 09:17:40,410 fail2ban.jail           [13806]: INFO    Creating new jail 'pam-generic'
2018-11-25 09:17:40,410 fail2ban.jail           [13806]: INFO    Jail 'pam-generic' uses systemd {}
2018-11-25 09:17:40,415 fail2ban.jail           [13806]: INFO    Initiated 'systemd' backend
2018-11-25 09:17:40,416 fail2ban.filter         [13806]: INFO    Set maxRetry = 2
2018-11-25 09:17:40,418 fail2ban.filter         [13806]: INFO    Set jail log file encoding to ISO-8859-1
2018-11-25 09:17:40,418 fail2ban.actions        [13806]: INFO    Set banTime = 3600
2018-11-25 09:17:40,419 fail2ban.filter         [13806]: INFO    Set findtime = 1600
2018-11-25 09:17:40,477 fail2ban.jail           [13806]: INFO    Jail 'sshd' started
2018-11-25 09:17:40,488 fail2ban.jail           [13806]: INFO    Jail 'sshd-ddos' started
2018-11-25 09:17:40,488 fail2ban.filtersystemd  [13806]: NOTICE  Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
2018-11-25 09:17:40,498 fail2ban.jail           [13806]: INFO    Jail 'proftpd' started
2018-11-25 09:17:40,514 fail2ban.jail           [13806]: INFO    Jail 'sendmail-auth' started
2018-11-25 09:17:40,520 fail2ban.jail           [13806]: INFO    Jail 'sendmail-reject' started
2018-11-25 09:17:40,560 fail2ban.jail           [13806]: INFO    Jail 'dovecot' started
2018-11-25 09:17:40,561 fail2ban.filtersystemd  [13806]: NOTICE  Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
2018-11-25 09:17:40,595 fail2ban                [13806]: CRITICAL Unhandled exception in Fail2Ban:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fail2ban/server/jailthread.py", line 66, in run_with_except_hook
  File "/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.py", line 263, in run
  File "/usr/lib64/python2.7/site-packages/systemd/journal.py", line 272, in wait
OSError: [Errno 24] Too many open files
2018-11-25 09:17:40,598 fail2ban.action         [13806]: ERROR   iptables -w -N f2b-pam-generic
iptables -w -A f2b-pam-generic -j RETURN
iptables -w -I INPUT -p tcp -j f2b-pam-generic -- failed with [Errno 24] Too many open files
2018-11-25 09:17:40,599 fail2ban.actions        [13806]: ERROR   Failed to start jail 'pam-generic' action 'iptables-allports': local variable 'stdout' referenced before assignment
2018-11-25 09:17:40,635 fail2ban.jail           [13806]: INFO    Jail 'pam-generic' started
2018-11-25 09:17:42,443 fail2ban                [13806]: CRITICAL Unhandled exception in Fail2Ban:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fail2ban/server/jailthread.py", line 66, in run_with_except_hook
  File "/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.py", line 263, in run
  File "/usr/lib64/python2.7/site-packages/systemd/journal.py", line 272, in wait
OSError: [Errno 24] Too many open files
2018-11-25 09:17:42,444 fail2ban                [13806]: CRITICAL Unhandled exception in Fail2Ban:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fail2ban/server/jailthread.py", line 66, in run_with_except_hook
  File "/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.py", line 263, in run
  File "/usr/lib64/python2.7/site-packages/systemd/journal.py", line 272, in wait
OSError: [Errno 24] Too many open files
2018-11-25 09:17:43,189 fail2ban                [13806]: CRITICAL Unhandled exception in Fail2Ban:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fail2ban/server/jailthread.py", line 66, in run_with_except_hook
  File "/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.py", line 263, in run
  File "/usr/lib64/python2.7/site-packages/systemd/journal.py", line 272, in wait
OSError: [Errno 24] Too many open files

Thanks and regards

Richard

More information about the Blueonyx mailing list