now browsing by category



Microsoft launches new Skype for Linux


Compared to the current version 7 for Skype on Windows, Microsoft (who bought Skype back in May 2011 for $8.5 billion) started neglecting the Linux desktop client – which is stuck on version 4.3, causing it to become outdated from a design perspective and not entirely functional. This was not a surprise considering Linux is a direct competitor for Microsoft’ Windows operating system.
The real surprise was the moment it was announced there’s a new version of Skype for Linux available. And this one is really new and different. These news shows that Microsoft is finally recognizing the importance of the Linux desktop, even though, for now, it represents only about 2.5% of share market.

Microsoft launched an alpha version of a new client for Linux on 13 July 2016, in a push to get users of the open-source operating system to make video calls and send messages with Skype.
As an alpha version, the new Skype for Linux has a number of drawbacks. For one, you can’t make or take calls to anyone using the previous versions of Skype for Linux (v4.3.0.37), and if you were hoping to be able to place video calls you’re similarly out of luck.

Things that are NOT supported in the Skype for Linux alpha:

  • Video calls
  • Calling telephone numbers
  • Sending SMS messages
  • Desktop screen sharing
  • Buying Skype credit
  • Adding new people to an existing call
  • Changing audio device settings
  • Support for running two clients at once
  • 32-bit Linux support

Things that are supported:

  • Instant messaging (including group instant messages)
  • One-to-one voice calls
  • Group voice calls
  • Add contacts
  • Use new emoticon packs
  • Call forwarding
  • Sharing contacts
  • Allows you to access 30 days of conversation history

Skype caution that the alpha release is ‘not a fully functioning Skype client as of yet’ and that more features will arrive in the near future. They’ve said video calling is a priority, and that they’ll be looking at ways to improve integration with Linux desktop features, such as native notifications. More information about available features you can find it on Skype’s website.

Another feature announced Wednesday was the alpha version of Skype based on WebRTC (web real-time communication is an open source initiative that lets internet users communicate in real time through voice and video — simply by using a Real-Time Communications (RTC) compatible browser — and negates the need to install plugins). Video calling and calls to landlines and mobiles are “coming soon” to Chrome browsers in Linux and sometime in the coming weeks and months to Chromebooks, officials said in a blog post. Skype for Web is running on Microsoft Edge without any plugin installation and it seems that the WebRTC Web-based version allows people to make calls ONLY in Google Chrome. You can check this out. If you open, authenticate with your user name, you will see the difference between the two browsers. In Firefox, you cannot make any calls yet, while this is possible in Chrome.
With the new Skype for Linux Alpha you can call people using the Skype clients for Windows, Mac, iOS, Android, and Skype for Web, but you won’t be able to call those who still use the old Skype for Linux. This is because the new app uses Skype’s “next generation calling architecture”, Microsoft says.

If you are a Linux user, you can download Skype for Linux Alpha DEB here and Skype for Linux Alpha RPM here. The DEB package has been tested on Ubuntu 16.04, Ubuntu Gnome 16.04, and Debian 8.5. The RPM package has been tested on Fedora 23, OpenSuse KDE 13.2, and OpenSuse Leap 42.1 KDE.

Surviving an XML-RPC Attack


XML-RPC is a remote procedure calling API using HTTP as the transport and XML as the encoding. This API is turned on by default in WordPress 3.5+. For a full list of the WordPress API functions available to developers via XML-RPC, take a look at the WordPress codex.

How do we know we were subject to an attack?

1. periodically having problem with the website being too slow (sometimes the DB server was even crashing)
2. finding many entries similar to “POST /xmlrpc.php HTTP/1.0” in our web server logs

In the last attack I saw live, they were flooding our server with POST requests to /xmlrpc.php from 16 different IP addresses.
The attack I witnessed lasted less than 10 minutes:
- Attack begins: 16:15:15
- Attack ends (ended by our defense mechanism): 16:24:51

During this time they sent more than 15,000 POST requests. You can see how many times each machine hit us in the left column.

[user@machine logs]$ cat com-access_log | awk '{print $1}' | sort -n | uniq -c | sort -nr

That many POST requests worries me a lot. We know already that one of the latest technique allows attackers to try a large number of WordPress username and password login combinations in a single HTTP request.

One way to mitigate this attack is by disabling the ability to call the “system.multicall” method in your WordPress installation by editing your functions.php file. Adding the function mmx_remove_xmlrpc_methods() will alleviate the problem, like so:

function mmx_remove_xmlrpc_methods( $methods ) {
    unset( $methods['system.multicall'] );
    return $methods;
add_filter( 'xmlrpc_methods', 'mmx_remove_xmlrpc_methods');

Using the whois tool I found out that the IPs used in attack belong to Leaseweb USA, Inc., a rather big Internet hosting service company, which, funny thing, is dedicated to fighting spam and cyber-crime (source).

Leaseweb has quite a big chunk of public IPs, 16,384 to be more precise (CIDR = And about 2048 for CIDR Who knows, maybe they have more.

Why are we being attacked?

I don’t know for sure, but my guess is just because our website seemed vulnerable. It is running an older version of the popular WordPress software.

How to prevent these attacks?

To prevent XML-RPC attacks, there are WordPress plugins one can add to the website, or we could manually block XML-RPC directly in the server configuration. If you are not using this XML-RPC API, maybe is best to disable the API for good.

The attack we faced was so powerful that was bringing the website to a halt, so we decided to also take actions to prevent any other types of DDoS (Distributed Denial of Service). This is why we looked into using mod_evasive for our Apache server that is powering the website.

mod_evasive provides evasive action in the event of an HTTP DoS attack or brute force attack. It is also designed to be a detection and network management tool, and can be easily configured to talk to firewalls, routers, and more. It can present reports abuse via email and syslog facilities.

Installation is pretty easy. The package should be directly in your distro repository. In our case, for Centos 7, we needed to enable the EPEL repository.

[user@machine ~]$ sudo yum install epel-release -y
[user@machine ~]$ sudo yum install mod_evasive -y
[user@machine ~]$ sudo systemctl restart httpd

After checking out the logs, and seeing a pattern in time interval used to rotate IPs, I modified the default configuration for mod_evasive. A few minutes later, this is what I can see in the httpd status:

[user@machine ~]$ sudo systemctl status -l httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-09-16 19:17:43 UTC; 9min ago
     Docs: man:httpd(8)
  Process: 3992 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
 Main PID: 4062 (httpd)
   Status: "Total requests: 32342; Current requests/sec: 81.8; Current traffic:  68KB/sec"
   CGroup: /system.slice/httpd.service
           ├─4062 /usr/sbin/httpd -DFOREGROUND
           ├─4063 /usr/sbin/httpd -DFOREGROUND
           ├─4064 /usr/sbin/httpd -DFOREGROUND
           └─4620 /usr/sbin/httpd -DFOREGROUND

Sep 16 19:18:04 machine mod_evasive[4065]: Blacklisting address possible DoS attack.
Sep 16 19:18:08 machine mod_evasive[4065]: Blacklisting address possible DoS attack.
Sep 16 19:18:08 machine mod_evasive[4066]: Blacklisting address possible DoS attack.
Sep 16 19:18:08 machine mod_evasive[4066]: Blacklisting address possible DoS attack.
Sep 16 19:18:10 machine mod_evasive[4066]: Blacklisting address possible DoS attack.
Sep 16 19:18:15 machine mod_evasive[4066]: Blacklisting address possible DoS attack.
Sep 16 19:18:15 machine mod_evasive[4066]: Blacklisting address possible DoS attack.
Sep 16 19:18:22 machine mod_evasive[4066]: Blacklisting address possible DoS attack.
Sep 16 19:18:22 machine mod_evasive[4066]: Blacklisting address possible DoS attack.
Sep 16 19:18:24 machine mod_evasive[4066]: Blacklisting address possible DoS attack.

Great success! The IPs where attack is coming from are starting to be blocked.
On my machine the configuration file is located at: /etc/httpd/conf.d/mod_evasive.conf.
Here is the configuration I am using:

    DOSHashTableSize    3097
    DOSPageCount        20
    DOSSiteCount        100
    DOSPageInterval     1
    DOSSiteInterval     1
    DOSBlockingPeriod   86400
    DOSSystemCommand    "/sbin/iptables -I INPUT -s %s -j DROP"

Some notes about our configuration:
- some of the values are the standard values; more info about what each one of these settings is doing can be found inside mod_evasive.conf (in the comments)
- usually a number set as value means that many seconds
- DOSPageCount and DOSSiteCount are two other parameters recommended to be changed to less aggressive values to avoid clients getting blocked unnecessarily
- DOSPageCount is the limit for the number of requests for the same page per page interval (set by DOSPageCount)
- DOSSiteCount is the limit for the total number of requests for the same website by an IP address per site interval (DOSSiteInterval)
- DOSBlockingPeriod is set for one week; it is the amount of time an IP address will be blocked for, if they are added to the blocked list.
- DOSWhitelist: we make sure we never block localhost
- DOSSystemCommand: we’re using a command that adds an iptable rule for blocking that particular IP forever

This is it. And by the way, we manually blacklisted all the Leaseweb’s known IP ranges. We don’t want their business.

Open Source Code Worth $5B


Back in 1976, Bill Gates published his “Open Letter to Hobbyists”, claiming that if software was freely shared it would prevent the writing of good software. He asked rhetorically, “Who can afford to do professional work for nothing? What hobbyist can put three man-years into programming, finding all bugs, documenting his product, and distribute it for free?” He presumed these were unanswerable questions, and both he and others based an industry on this assumption. Now, however, there are thousands of developers who are writing their own excellent code, and then giving it away. Gates was fundamentally wrong: sharing source code, and allowing others to extend it, is indeed a practical approach to developing large-scale systems – and its products can be more reliable.

Richard Stallman who had a totally different opinion, launched the GNU Project in September 1983 and was the pioneer of what we call copyleft (a play on the word copyright) is the practice of offering people the right to freely distribute copies and modified versions of a work with the stipulation that the same rights be preserved in derivative works down the line.

Later on, in 1991, Linux Torvalds chose to use one of these permissive free software licences (GNU General Public License or GPL) for his new hobby project.

Fast forward today, after more than 25 years, The Linux Foundation, a foundation built in 2007 around Linus’s project (the Linux kernel), and derivatives, estimates that more than 500 companies and thousands of developers from around the world contribute to these open source software projects.

For example only Red Hat Linux 7.1 includes over 30 million physical source lines of code (SLOC), compared to well over 17 million SLOC in version 6.2 (which had been released about one year earlier). Using the COCOMO cost model, this system is estimated to have required about 8,000 person-years of development time (as compared to 4,500 person-years to develop version 6.2).

Here are some key findings from the white paper that the Linux Foundation released at the end of September 2015 (source):

  • The total lines of source code present today in Linux Foundation’s Collaborative Projects are 115,013,302.
  • The estimated, total amount of effort required to retrace the steps of collaborative development for these projects is 41,192.25 person years.
  • In other words, it would take 1,356 developers 30 years to recreate the code bases present in Linux Foundation’s current Collaborative Projects.
  • The total economic value of this work is estimated to be more than $5 billion dollars.

Linux conquered the world without anyone noticing. And is all because of using the power of many. Even Microsoft, Bill Gates gigantic corporation “loves” linux. It’s a hate/love relationship spanning decades, but they are not ignoring it, not for a long time:

  • Microsoft Loves Linux because Linux powers Azure networking, which is their most promising business unit. (source)
  • Microsoft used Linux to power their home page to protect against DOS attacks. (source)
  • Microsoft paid over $25 billion to buy a $3 billion revenue division running almost entirely on Linux. (source)
  • Linux servers used to centralize all the Skype traffic after the acquisition. (source)

Difference Between FTPS and SFTP


Recently we were in the process of setting up remote file transfer capabilities for some applications, and we wanted to be as simple and secure as possible.

With that in mind, which is better, FTPS or SFTP?

They are two completely different protocols. FTPS is FTP with SSL for security. It uses a control channel and opens new connections for the data transfer. As it uses SSL, it requires a certificate.

SFTP (SSH File Transfer Protocol/Secure File Transfer Protocol) was designed as an extension of SSH to provide file transfer capability, so it usually uses only the SSH port for both data and control. In most SSH server installations you will have SFTP support, but FTPS would need the additional configuration of a supported FTP server.

In order to address this issue a set of security extensions to the original FTP protocol were proposed in RFC 2228 that protect FTP data as it travels over the network using SSL encryption. FTPS (FTP/SSL) is a name used to provide a number of ways that FTP software can perform secure file transfers. Each way involves the use of a SSL/TLS layer below the standard FTP protocol to encrypt the control and/or data channels.

Secure variants of FTP include FTPS Implicit SSL and FTPS Explicit SSL.  Both utilize SSL encryption.

FTPS Implicit SSL

In implicit SSL mode a required SSL session is established between client and server before any data is exchanged.  As it’s name suggests, the use of SSL is implied and any connection attempt made by a client without using SSL are refused by the server.  FTPS implicit SSL services generally run on port 990.   Although still in use today, FTPS Implicit SSL is considered by many to be obsolete in favor of FTPS Explicit SSL.

FTPS Explicit SSL

In explicit SSL mode the client and server negotiate the level of protection used.  This is very useful in that the server can support both unencrypted FTP and encrypted FTPS sessions on a single port.  In an explicit SSL session the client first establishes an unencrypted connection to the FTP service.  Prior to sending user credentials, the client requests that the server switch the command channel to an SSL encrypted channel by sending the AUTH TLS or AUTH SSL command.  Upon successful setup of the SSL channel the client sends user credentials to the FTP server.  These credentials along with any other commands sent to server during the FTP session are automatically encrypted by the SSL channel.  Similar to the way in which the command channel may be protected, the level of protection used on the data channel is negotiated between the client and server using the PROT command.

FTPS Pros:

  • Widely known and used
  • The communication can be read and understood by a human
  • Provides services for server-to-server file transfer
  • SSL/TLS has good authentication mechanisms (X.509 certificate features)
  • FTP and SSL/TLS support is built into many internet communications frameworks

FTPS Cons:

  • Does not have a uniform directory listing format
  • Requires a secondary DATA channel, which makes it hard to use behind firewalls
  • Does not define a standard for file name character sets (encodings)
  • Not all FTP servers support SSL/TLS
  • Does not have a standard way to get and change file or directory attributes

Firewall permissions

Server -  Allow inbound connections on port 21 and / or 990.  Define passive port range (e.g. 2000-2500) for file transfers and directory listings and allow inbound connections on passive port range.  Consult your server documentation for instructions on how to set a passive port range.

Client – Allow outbound connections to port 21 and passive port range defined by server.

SFTP is often confused with FTPS and vice-versa even though these protocols share nothing in common except their ability to securely transfer

The SFTP (SSH File Transfer Protocol), also known as the Secure File Transfer Protocol, enables secure file transfer capabilities between networked hosts. Unlike the Secure Copy Protocol (SCP), SFTP additionally provides remote file system management functionality, allowing applications to resume interrupted file transfers, list the contents of remote directories, and delete remote files. All data sent between client and server is encrypted using an agreed upon encryption cipher. SFTP sessions can also be further protected through the use of public and private keys, which offer an alternative form of authentication known as public key authentication. This can be used as an alternative to or in conjunction with the traditional form of authentication of usernames and passwords.

SFTP Pros:

  • Has a good standards background which strictly defines most (if not all) aspects of operations
  • Has only one connection (no need for a DATA connection)
  • The connection is always secured
  • The directory listing is uniform and machine-readable
  • The protocol includes operations for permission and attribute manipulation, file locking, and more functionality

SFTP Cons:

  • The communication is binary and can not be logged “as is” for human reading
  • SSH keys are harder to manage and validate
  • The standards define certain things as optional or recommended, which leads to certain compatibility problems between different software titles from different vendors.
  • No server-to-server copy and recursive directory removal operations
  • No built-in SSH/SFTP support in VCL and .NET frameworks

Firewall permissions
Server – Allow inbound connections on port 22.
Client – Allow outbound connections to port 22.
One of the most noticeable differences between SSL/TLS and SSH is that SSL normally (yes, there can be exceptions) employs X.509 digital certificates for server and client authentication whereas SSH does not. And because SSL uses digital certificates, it consequently also requires the presence of a public key infrastructure (PKI) and the participation of a certificate authority (CA).

Although it’s possible to employ what are known as self-signed certificates, in which case SSL then becomes very similar to SSH due to the absence of a CA, this is not a recommended practice. Self-signed certificates are only acceptable in intra-organizational transactions, except in large (e.g. globally distributed) organizations.

Another big difference is that SSH has more functionality built into it. For instance, on its own, SSH can enable users to login to a server and execute commands remotely. SSL does not have this capability. You would need to pair it with another protocol (e.g. HTTP, FTP, or WebDAV) in order for it to have similar functions.

SSH also readily supports connection multiplexing, flow control, terminal management, and other features. Of course, those additional features no longer fall under our original discussion wherein we started comparing SSL and SSH in relation to SFTP and FTPS.

Running tomcat as a systemd unit


In a previous blog (see What is systemd), I talked about the new init system that most linux distributions are switching to. You can go read that one first if some of the things we talk about here are not making much sense.
Centos 7 our OS of choice comes with Tomcat 7.0.57. To take advantage of all the improvements and bug fixings done lately we wanted to have the latest version of Tomcat 7 installed on our system. (version 7.0.65 at the moment of writing this).

We downloaded this version, installed it, setup our applications, etc. Everything went smooth. There was one thing we started missing once we migrate to our new setup. The ability to run Tomcat at system boot, and the ability to just use systemd’s main command systemctl to start/stop/status etc.

So this is what we did to have downloaded Tomcat7 instance controlled via systemd.
First we created a new systemd service unit file "/etc/systemd/system/tomcat7.service". A systemd’s “service unit” describes how to manage a service or application on the server. This will include how to start or stop the service, under which circumstances it should be automatically started, and the dependency and ordering information for related software.
The available unit files for systemd can be seen in /usr/lib/systemd/system/ and /etc/systemd/system/ (the latter takes precedence). Notice that our new tomcat7 unit file is located in one of these special folders.

The list of installed unit files in your system can be seen using the following command. At the end of the article your list will include a new one called “tomcat.service”

[user@machine ~]$ systemctl list-unit-files

Here is the content of our tomcat7.service file

[user@machine ~]$ cat /etc/systemd/system/tomcat7.service
Description=Apache Tomcat 7 Servlet Container

Let’s explain some of the content of this file. First of all, the syntax: the content is organized in sections, which are denoted by a pair of square brackets with the section name (case sensitive) enclosed within. Each section extends until the beginning of the subsequent section or until the end of the file. So, in our case we have 3 sections: Unit, Service and Install; Within these sections, unit behavior and metadata is defined through the use of simple directives using a key-value format with assignment indicated by an equal sign. If a ceratin pair is not specified than, the systemd’s default value for that specific key is going to be used.

The [Unit] section is generally used for defining metadata for the unit and configuring the relationship of the unit to other units. In here, first we are setting the unit description, see Description= directive.
The units listed in the After= directive will be started before starting the current unit. So, in our example we are making sure that the syslog and network targets are started before the tomcat7 service. This directive does not imply a dependency relationship (like the “Requires=” or “Wants=” does).

The [Service] section is used to provide configuration that is only applicable for services. Here we specify the OS user and group to be used as owner of the process spawned (see User= and Group= pairs). The Type= directive being used is “forking”, which means the service forks a child process, exiting the parent process almost immediately. This tells systemd that the process is still running even though the parent exited.
ExecStart= specifies the full path of the command to be executed to start tomcat, while ExecStop= specifies the full path of the command to be executed to stop tomcat. (If this is not given, the process will be killed immediately when the service is stopped.)
Restart= This indicates the circumstances under which systemd will attempt to automatically restart the service. This can be set to values like “always”, “on-success”, “on-failure”, “on-abnormal”, “on-abort”, or “on-watchdog”. These will trigger a restart according to the way that the service was stopped.
systemd has Environment directive which sets environment variables for executed processes. In our case we set the Tomcat’s CATALINA_PID, CATALINA_HOME and CATALINA_BASE.

The WantedBy= directive in the [Install] section is the most common way to specify how a unit should be enabled. Specifying the is the systemd equivalent of runlevel 3, which is a multi-user system that boots to a console, not a GUI.

After having this tomcat7.service file in place you have to notify systemd that a new name.service file exists by executing the following command as root:

[user@machine ~]$ systemctl daemon-reload

Once this is done you can use systemctl to start, stop or enable tomcat7 service.

[user@machine ~]$ systemctl start name.service
[user@machine ~]$ systemctl stop name.service

The output of our enable command is:

[user@machine ~]$ $ sudo systemctl enable tomcat7.service
Created symlink from /etc/systemd/system/ to /etc/systemd/system/tomcat7.service.

Enabling an unit means it will be started on system bootup, Just to test this is really happening, we restarted our machine and we checked the status of our new tomcat7.service

[user@machine ~]$ $ sudo systemctl status tomcat7.service
tomcat7.service - Apache Tomcat 7 Servlet Container
Loaded: loaded (/etc/systemd/system/tomcat7.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2016-01-10 10:30:44 EST; 1min 48s ago
Process: 797 ExecStart=/opt/apache-tomcat-7.0.65/bin/ (code=exited, status=0/SUCCESS)
Main PID: 822 (java)
CGroup: /system.slice/tomcat7.service
└─822 java -Djava.util.logging.config.file=/opt/apache-tomcat-7.0.65/conf/ -Djava.util....

Jan 10 10:30:44 staging-ods systemd[1]: Starting Apache Tomcat 7 Servlet Container...
Jan 10 10:30:44 staging-ods systemd[1]: Started Apache Tomcat 7 Servlet Container.

Great success! We’re done.