Background:
In today’s current environment it’s not unusual to see log files filled with thousands of entries for failed logins. Botnets, compromised servers, and even foreign governments are directing their energies toward harvesting valid username/passwords for mail servers, SSH access, and web sites with massive dictionary and brute force attacks. These attacks can come from several different sources at once and it’s not unheard of to see multiple machines co-ordinating a dictionary attack against a target machine.
To provide the reader with an idea of the scale, we have seen server log files for failed SSH logins grow to 27mb in size and accounted for more 40,000 login failures in a single week. That was for a small mom and pop web site that had no eCommerce, no forum, or any other obvious attractive targets on it. Clients with larger, more prominent services that have attracted a botnet’s full attention can see their log files grow into the hundreds of mb in no time. Imagine the number of attacks being directed against banking, social networking, and other high profile entities on the Internet.
There are several strategies for defeating this type of attack, both active and passive. The most common of the passive counter measures is to employ very strong passwords, or move the attacked service to a different port. These methods won’t work for a lot of businesses for several reasons. Moving a web or email service to a non standard port increases support costs and often breaks software. The general public notoriously picks weak passwords, and even when warned that their passwords are weak, use them anyway. (How to create a secure password)
This common problems has led to a couple of active solutions that work by monitoring log files and then executing commands to block a host based on behavior extracted from logs. The most prominent is Fail2ban, an open source package written in Python, that can utilize several different methods to block the offending host. The most common methods for blocking hosts in the Linux/BSD world is to use iptables or ipfw rules.
The Issues:
Fail2ban works very well for the individual server, but most large internet service deployments are comprised of several servers behind a firewall. The problem we were faced with was not just the large number of physical machines, but the fact that each host had a number of virtual hosts that needed protection as well. We wanted to block an attacking machine from all protected servers once it has violated the policy on any one of them.
If we fell back on just installing Fail2ban on each server, then a offending host is only blocked from one server at a time while still having access to the others. In a large virtual environment where one is trying to keep the VM’s as free of extraneous software as possible, having iptables installed on each virtual host chews into memory requirements. Managing the Fail2ban infrastructure across all machines becomes more complex and more costly as the number of machines increases.
The Solution:
In this case we have a Linux based firewall between the Internet and the servers. Each machine and VM behind the firewall runs Fail2ban, but instead of creating a local rule, each host sends the IP of the bad host to the firewall. This firewall will be used to block hosts for the entire network. We accomplish this by using Fail2ban’s client program and some custom configuration on the protected hosts and firewall. Below is an outline of how to centralize the blocking of offending host’s IP address using a Linux firewall, iptables, and Fail2ban.
Setting up the firewall:
Install Fail2ban on the firewall (For RHEL and clones, yum install fail2ban, for Debian apt-get install fail2ban). Edit the /etc/fail2ban/jail.conf. Adjust ignoreip to include IPs that should not be banned for any reason ever, this usually means management and monitoring networks. Add a new jail as follows;
[default-iptables]enabled = truefilter = defaultaction = iptables[name=default, port=ssh, protocol=all]sendmail-whois[name=default, dest=someone@geeforce.net, sender=fail2ban@fw.clientdomain.com]logpath = /var/log/fail2banmaxretry = 1
In our case no other jails are relevant on the firewall, so they have all been set to “enabled = false”. Now we edit the defaults action-iptables.conf file found in /etc/fail2ban/action.d
[Definition]
actionstart = iptables -N fail2ban-<name>
iptables -A fail2ban-<name> -j RETURN
iptables -I FORWARD -p <protocol> -j fail2ban-<name>
actionstop = iptables -D FORWARD -p <protocol> -j fail2ban-<name>
iptables -F fail2ban-<name>
iptables -X fail2ban-<name>
actioncheck = iptables -n -L FORWARD | grep -q fail2ban-<name>
actionban = iptables -I fail2ban-<name> 1 -s <ip> -j DROP
actionunban = iptables -D fail2ban-<name> -s <ip> -j DROP[Init]
name = default
port = default
protocol = all
Finally we set up a chroot jail for a user on the firewall (Fail2ban) that has sudo permissions to run 2 commands /usr/bin/fail2ban-client and /bin/touch /var/log/fail2ban. Be sure to touch /var/log/fail2ban and ensure that it is owned by the Fail2ban user. This is necessary because the Fail2ban client doesn’t work the way you might think, and the user has to be able to touch that file to update the date/timestamp. Create a ssh key pair to allow logins from remote hosts.
On the hosts, install Fail2ban and edit the /etc/fail2ban/jail.conf file. Add the following jail;
[ssh-rban]
enabled = true
filter = sshd
action = rban[name=SSH, fwip=IP.OF.FW.HERE]
logpath = /var/log/secure
maxretry = 2
Create the file /etc/fail2ban/action.d/action-rban.conf that looks like the following
[Definition]
actionstart =
actionstop =
actioncheck = touch /var/log/fail2ban
actionban = /usr/bin/ssh -v -l fail2ban -tt <fwip> ‘sudo /usr/bin/fail2ban-client set default-iptables banip <ip>’ &&
/usr/bin/ssh -v -l fail2ban -tt <fwip> ‘/bin/touch /var/log/fail2ban’;
actionunban =
Test ssh connectivity by logging in to with fail2ban user from your host as root (fail2ban usually runs as root). Once you’ve succeeded in logging in, start Fail2ban on both machines and test. This example only checks SSH logs for failed logins. You can modify this setup as a template for any of the checks available for Fail2ban using the same action. This includes denial of service attacks, pop3 login failures, web login failures, and more.
Security Considerations
The methodology employed here took into account several different security and management considerations. One of the more common variations of this theme is to allow the remote Fail2ban server to add an iptables rule on the firewall directly. The method was rejected outright because providing the ability for a host to directly manipulate the firewall rules is extremely problematic from a security standpoint. In the arrangement outlined here, a protected host could only block other hosts using the Fail2ban client. Even the ability to unban clients is purposely disallowed. The firewall itself makes the decisions on when to unban an IP.
An attacker that has gained entry to a protected host cannot override the centralize white list, nor does the firewall user allow an attacker to directly manipulate iptables. The Fail2ban user on the firewall is specifically setup to be highly unprivileged, and limited, especially with a chroot jail in place. The firewall essentially doesn’t trust the servers it protects.
This arrangement centralizes the management of all whitelisted IPs. That makes changes very easy to roll out. One caveat on the arrangement outline here; this has been specifically setup to block the offending IP from reaching ANY port on ANY machine for the duration of the ban time (defined in /etc/fail2ban/jail.conf). That was a design choice, but the Fail2ban setup lends itself to banning specific service from offending hosts as opposed to banning all services. A denial of service can result from innocent people tripping the Fail2ban actions. In this situation, that risk was considered minimal in regards to the amount of protection provided.
Conclusions;
Using Fail2ban to centralize the blocking of hosts has been a success everywhere it’s been deployed. In one cluster alone, the typical /var/log/secure files went from being 10-20Mb or more every week, to a mere 200K. Tens of thousands of password fails have been replaced by as little as 15. By turning on different filters, one is also able to catch dictionary and brute force attacks against SMTP servers, WordPress sites, and webmail installations and protect them all with the same firewall. This solution isn’t limited to the Linux or even Unix world. With a centralized log server one can use Fail2ban to protect mixed environments, including Windows and Mac servers and centralize administration even more. Fail2ban and an active centralized response to attacks just scratches the surface of a fully fleshed out security infrastructure.
