Surviving an XML-RPC Attack

IT

XML-RPC is a remote procedure calling API using HTTP as the transport and XML as the encoding. This API is turned on by default in WordPress 3.5+. For a full list of the WordPress API functions available to developers via XML-RPC, take a look at the WordPress codex.

How do we know we were subject to an attack?

1. periodically having problem with the website being too slow (sometimes the DB server was even crashing)
2. finding many entries similar to “POST /xmlrpc.php HTTP/1.0” in our web server logs

In the last attack I saw live, they were flooding our server with POST requests to /xmlrpc.php from 16 different IP addresses.
The attack I witnessed lasted less than 10 minutes:
- Attack begins: 16:15:15
- Attack ends (ended by our defense mechanism): 16:24:51

During this time they sent more than 15,000 POST requests. You can see how many times each machine hit us in the left column.

[user@machine logs]$ cat com-access_log | awk '{print $1}' | sort -n | uniq -c | sort -nr
   2907 207.244.75.121
   2867 207.244.103.165
   2811 207.244.104.217
   2799 207.244.68.54
   2757 198.7.63.69
   2746 198.7.57.134
   2738 207.244.105.144
   2693 198.7.57.143
   2692 207.244.103.151
   2687 207.244.103.144
   2682 207.244.103.122
   2677 207.244.105.179
   2655 207.244.105.178
   2618 207.244.68.238
   2598 207.244.103.141
   2515 207.244.105.146

That many POST requests worries me a lot. We know already that one of the latest technique allows attackers to try a large number of WordPress username and password login combinations in a single HTTP request.

One way to mitigate this attack is by disabling the ability to call the “system.multicall” method in your WordPress installation by editing your functions.php file. Adding the function mmx_remove_xmlrpc_methods() will alleviate the problem, like so:

function mmx_remove_xmlrpc_methods( $methods ) {
    unset( $methods['system.multicall'] );
    return $methods;
}
add_filter( 'xmlrpc_methods', 'mmx_remove_xmlrpc_methods');

Using the whois tool I found out that the IPs used in attack belong to Leaseweb USA, Inc., a rather big Internet hosting service company, which, funny thing, is dedicated to fighting spam and cyber-crime (source).

Leaseweb has quite a big chunk of public IPs, 16,384 to be more precise (CIDR = 207.244.64.0/18). And about 2048 for CIDR 198.7.56.0/21. Who knows, maybe they have more.

Why are we being attacked?

I don’t know for sure, but my guess is just because our website seemed vulnerable. It is running an older version of the popular WordPress software.

How to prevent these attacks?

To prevent XML-RPC attacks, there are WordPress plugins one can add to the website, or we could manually block XML-RPC directly in the server configuration. If you are not using this XML-RPC API, maybe is best to disable the API for good.

The attack we faced was so powerful that was bringing the website to a halt, so we decided to also take actions to prevent any other types of DDoS (Distributed Denial of Service). This is why we looked into using mod_evasive for our Apache server that is powering the website.

mod_evasive provides evasive action in the event of an HTTP DoS attack or brute force attack. It is also designed to be a detection and network management tool, and can be easily configured to talk to firewalls, routers, and more. It can present reports abuse via email and syslog facilities.

Installation is pretty easy. The package should be directly in your distro repository. In our case, for Centos 7, we needed to enable the EPEL repository.

[user@machine ~]$ sudo yum install epel-release -y
..................................................
[user@machine ~]$ sudo yum install mod_evasive -y
..................................................
[user@machine ~]$ sudo systemctl restart httpd

After checking out the logs, and seeing a pattern in time interval used to rotate IPs, I modified the default configuration for mod_evasive. A few minutes later, this is what I can see in the httpd status:

[user@machine ~]$ sudo systemctl status -l httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-09-16 19:17:43 UTC; 9min ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 3992 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
 Main PID: 4062 (httpd)
   Status: "Total requests: 32342; Current requests/sec: 81.8; Current traffic:  68KB/sec"
   CGroup: /system.slice/httpd.service
           ├─4062 /usr/sbin/httpd -DFOREGROUND
           ├─4063 /usr/sbin/httpd -DFOREGROUND
           ├─4064 /usr/sbin/httpd -DFOREGROUND
           └─4620 /usr/sbin/httpd -DFOREGROUND

Sep 16 19:18:04 machine mod_evasive[4065]: Blacklisting address 207.244.68.54: possible DoS attack.
Sep 16 19:18:08 machine mod_evasive[4065]: Blacklisting address 198.7.57.134: possible DoS attack.
Sep 16 19:18:08 machine mod_evasive[4066]: Blacklisting address 207.244.103.141: possible DoS attack.
Sep 16 19:18:08 machine mod_evasive[4066]: Blacklisting address 198.7.63.69: possible DoS attack.
Sep 16 19:18:10 machine mod_evasive[4066]: Blacklisting address 207.244.103.151: possible DoS attack.
Sep 16 19:18:15 machine mod_evasive[4066]: Blacklisting address 198.7.57.143: possible DoS attack.
Sep 16 19:18:15 machine mod_evasive[4066]: Blacklisting address 207.244.75.121: possible DoS attack.
Sep 16 19:18:22 machine mod_evasive[4066]: Blacklisting address 207.244.105.179: possible DoS attack.
Sep 16 19:18:22 machine mod_evasive[4066]: Blacklisting address 207.244.68.238: possible DoS attack.
Sep 16 19:18:24 machine mod_evasive[4066]: Blacklisting address 207.244.103.144: possible DoS attack.

Great success! The IPs where attack is coming from are starting to be blocked.
On my machine the configuration file is located at: /etc/httpd/conf.d/mod_evasive.conf.
Here is the configuration I am using:

    DOSHashTableSize    3097
    DOSPageCount        20
    DOSSiteCount        100
    DOSPageInterval     1
    DOSSiteInterval     1
    DOSBlockingPeriod   86400
    DOSEmailNotify      radu.candea@neemeekaa.com
    DOSSystemCommand    "/sbin/iptables -I INPUT -s %s -j DROP"
    DOSWhitelist        127.0.0.1

Some notes about our configuration:
- some of the values are the standard values; more info about what each one of these settings is doing can be found inside mod_evasive.conf (in the comments)
- usually a number set as value means that many seconds
- DOSPageCount and DOSSiteCount are two other parameters recommended to be changed to less aggressive values to avoid clients getting blocked unnecessarily
- DOSPageCount is the limit for the number of requests for the same page per page interval (set by DOSPageCount)
- DOSSiteCount is the limit for the total number of requests for the same website by an IP address per site interval (DOSSiteInterval)
- DOSBlockingPeriod is set for one week; it is the amount of time an IP address will be blocked for, if they are added to the blocked list.
- DOSWhitelist: we make sure we never block localhost
- DOSSystemCommand: we’re using a command that adds an iptable rule for blocking that particular IP forever

This is it. And by the way, we manually blacklisted all the Leaseweb’s known IP ranges. We don’t want their business.