Using Fail2Ban To Control Access To Docker Applications

by Michael Bagnall
hello@flyingflip.com

Fail2Ban is one of the old school open-source intrusion prevention systems that is designed to provide additional layers of security for Unix based environments, It works by scanning log files for suspicious patterns or failed login attempts and then takes action by banning the IP address associated with the activity. Fail2Ban can be configured to monitor various services and protocols, including SSH, http/https, and FTP, among a host of others. You can find a complete list of out of the box services at the end of this article. Potential attacks are determined by multiple failed logins, bad page requests, or unauthorized attempts to access restricted resources. Fail2Ban can automatically update firewall rules to block the offending IP for a specified period. By default, Fail2Ban works in conjunction with IPTables or IPChains to block malicious activity at the operating system level.

By automatically managing bans and alerting administrators, it reduces the risk of exploitation while minimizing manual intervention. The benefits include better protection against brute-force attacks, reduced server load and greater flexibility through customizable rules. Fail2Ban is widely used in environments where security is critical but full-fledged intrusion detection systems (IDS) or web application firewalls (WAF) may be cost prohibitive or more than is needed for the security of the system.

Our use case is to make Fail2Ban available within a containerized, Docker based service. In this way intrusion detection can be detected on a specific service (in our case, a web service) and manage bans to the service without affecting other services in other containers. This is primarily to prevent issues such as false positives from keeping administrators out of systems and not to prohibit access to other services or applications for the affected IP. Put simply, in a shared hosting environment, one ban does not affect the other hosts on the shared system.

Our Specific Use Case
Our use case is to protect web traffic in a container. By this if you hit pages for which you do not have access or other pages that are outside the document root (.htaccess, .env, passwd and other files in the operating system level) then it will block the IP address of the actor attempting to perform this activity with the goal being to make it annoying or impossible enough for them to move onto an easier target and leave you alone. It will also serve to block any access which produces 403 errors. More errors could be included such as a flood of 404 errors or other kinds of web access or Unauthorized requests (401 errors). Even 200 replies – successful responses – could trigger the rules if the number of requests for resources exceeds a certain threshold within a specified period of time.

For this example, we’re going to create a rule that if you hit a 403 page more than 5 times in a one minute period, your IP will be blocked for one hour.

Preparing the Virtual Machine

In all fairness, little needs to be done to the host virtual machine that will contain your app services. You will need a proxy such as Nginx or Apache in proxy mode to route traffic from your host to the containers in question. But because Fail2Ban will operate at the container level and not the virtual machine level, there is no need to install Fail2Ban or IPTables on the host machine unless you wish to have global protection for SSH access or other forms of intrusion that would make sense for a host machine. Keep in the mind the goal of our demonstration here is to limit connections inside the service and not to the global host. There are many articles out there that cover host protection, so we’ll keep this specific to our goal.

The Service Container

When building a web application, there are normally two components. One is a base container that has all of your common software such as operating system, PHP, Apache, NodeJS or any other technology that will work to power a universal set of web sites. Individual applications can build on this container to add app specific components and perform build steps such as composer actions or SASS compiling.

It is in this container that we will install and configure our Fail2Ban system. You can access the sample repository on GitHub for all of the Dockerfile and configuration file examples. Sample GitHub Repository. You can also check out the container this creates at: Fail2Ban Docker Example.

The base container can contain pretty much anything you want it to have. For Fail2Ban to work you must have the following components inside your container.

IPTables
Fail2Ban
Syslogd

In your Dockerfile you would want to include (for Ubuntu systems).

RUN apt-get install -u fail2ban inetutils-syslogd iptables

To install the necessary dependencies,. Other Linux flavors will require different dependency names and maybe different package managers.

These three components are required for the Fail2Ban service to properly function and many flavors of Linux do not come with them pre-installed. I was surprised to find out that Syslogd is not installed by default on many flavors of Linux. This includes the Debian/Ubuntu distribution which is the example in this document. You can see an example for Alpine in the GitHub repository.

Understanding Docker Networking

In order to properly present the reasons for why we handle Fail2Ban in containers in the manner in which we are, it is important to provide some context on Docker container networking especially as it relates to the host virtual machine.

By default, Docker operates in a virtualized network environment. Even in Linux. Most configurations you will see are that the Docker containers are assigned a network name and the IPs that are used are mapped inside that network. You can then specify port mappings that map the port from your host VM to the relevant IP inside the container.

For Example
A web service container operates on ports 80/443. If you run multiple containers on a single VM, you must map these ports on different virtual networks since you can only run one service per port on a VM.
Given this, you would map a port such as 8080/8443 to ports 80/443 respectively and use a reverse proxy such as Nginx to route traffic from port 80/443 on your server to the container with the web site for that specific Virtual Host. In the case of Nginx, your server block.

So if you set up Fail2Ban in your virtualized network container and have your reverse proxy pass the IP of the traffic to your site through to the container, it will add to IPTables and should block traffic right? But not so fast. While the correct offending IP is added to IPTables, it is NOT blocked by the container. This is because your network traffic is coming from the IP of the host VM and not the requestor. While your proxy will pass this through to the other web service on the container, it does NOT pass it on the operating system level, just the service within the container.

IPTables works at the operating system level and not the container level. So if the network IP of your host (inside Docker) is 172.10.1.10 this is what IPTables inside the container will be looking for in terms of blocking, not the IP passed through on the 80/443 port. Even though you may pass through the request from the proxy to pass the user’s real IP, and the web server will respect it, the container itself will block it on the OS level. While this would be successful in blocking every request VM to the container – it would block it for everyone on the web ports since your traffic has to travel through the VM not just the offending traffic.

Conversely if you use the host network mode, the container connects directly to the same network as your VM. This is important for IP pass through to your container because this is how Fail2Ban works. It works by placing entries at the operating system level as opposed to the web service label. For it to work, Fail2Ban needs to be able to add IPs to the IPTables firewall in a way that the host container will respect them inside the container and block the access.

Lost yet? Let’s take a look at some examples in terms of using a *docker-compose.yml* file.

services: webapp: image: your/image container_name: container_name ports: - "8097" - "9097" network: - network_name restart: unless-stoppes networks: network_name:

IPTables with Host Networking Mode Enabled
services: webapp: image: your/image container_name: container_name cap_add: - NET_ADMIN expose: - "8097" - "9097" network_mode: host restart: unless-stopped

How To Configure The Fail2Ban Service in Your Container

There are a lot of ways to configure Fail2Ban in your container and some come out of the box. However for our use case, we wish to block floods of 403 errors. In our rule, we block the IP if someone attempts to access a 403 page more than 10 times in a minute. You can configure your rules to be more strictly or lax depending on your use case.

To do this, you will need to add two files to the default configuration in /etc/fail2ban. The first is a filter file which defines how the traffic is to be filtered. The second is the jail file which defines the parameters of the service to look at, and the rules for blocking and releasing IP addresses.

Here are the two examples we use:
/etc/fail2ban/filter.d/httpd-forbidden.conf
[Definition] failregex = ^ - - .*HTTP/[0-9]+(.[0-9]+)?" 403 ^ -.*"(GET|POST|HEAD).*HTTP.*"/[0-9]+(.[0-9]+)?" 403

#### /etc/fail2ban/jail.d/http-forbidden.conf
[httpd-forbidden] enabled = true filter = httpd-forbidden backend = polling logpath = /var/log/apache2/*access.log

bantime = 1h maxretry = 5 findtime = 500 port = http,https banaction = iptables-multiport action = iptables-multiport[name=apache, port="http,https"]

Adding these Persistently in Docker

While you could, in theory, mount these as volumes in your container, it is sometimes best on containers to include it in your Dockerfile as part of your build process. The latter is the approach we have taken in our example repositories. We will present the alternative method of mounting configuration files later. But remember, as with any mounted folder, there are special considerations for the things that must be kept in mounted volumes such as pre-existing files that may not need configuration. With this there are pros and cons as well. All in due course.

Dockerfile configuration of fail2ban configuration
Place the files in the etc/fail2ban/filter.d and etc/fail2ban/jail.d of your container directory as specified in the copy commands below or modify those paths to point to the correct location of those files.

# This fail2ban configuration took forever to get right because of the constantly # changing versions of fail2ban and apache # # https://talk.plesk.com/threads/block-repeated-403-forbidden-requests-with-fail2ban-as-included-plesk-jail-feature-request.358827/ COPY etc/fail2ban/filter.d/httpd-forbidden.conf /etc/fail2ban/filter.d/httpd-forbidden.conf COPY etc/fail2ban/jail.d/httpd-forbidden.conf /etc/fail2ban/jail.d/httpd-forbidden.conf