Security hardening on Ubuntu Server 14.04

Recently I’ve been involved with a project where I needed to perform some security hardening on Amazon Web Services EC2 instances running Ubuntu Server 12.04, so I used this excellent guide as a starting point, then I added, removed and modified things as needed.

I decided to take those procedures and modify them for Ubuntu Server 14.04 now that this new LTS version has been released. Some of the procedures from 12.04 no longer need to be performed, and some needed to be changed. The following guidelines are what I’ve ended up with. You might find these guidelines useful to varying extents on other Linux distributions, but there will be potentially very significant differences depending on which distro you’re using.

Assume that all these operations need to be performed as root, which you can either do with sudo or by logging in as root first. (I’ve noticed that Ubuntu users seem particularly averse to logging in as root, apparently preferring instead to issue an endless series of commands starting with sudo, but I’m afraid that kind of extra hassle is not for me, so I just log in as root first.)

Harden SSH

I generally regard it as a very sensible idea to disable any kind of root login over SSH, so in /etc/ssh/sshd_config change PermitRootLogin to no.

If SSH on your servers is open to the world then I also advise running SSH on a non-standard port in order to avoid incoming SSH hacking attempts. To do that, in /etc/ssh/sshd_config change Port from 22 to another port of your choice, e.g. 1022. Note that you’ll need to update your firewall or EC2 security rules accordingly.

After making changes to SSH, reload the OpenSSH server:

service ssh reload

Limit su access to administrators only

It generally seems like a sensible idea to make sure that only users in the sudo group are able to run the su command in order to act as (or become) root:

dpkg-statoverride --update --add root sudo 4750 /bin/su

Improve IP security

Add the following lines to /etc/sysctl.d/10-network-security.conf to improve IP security:

# Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Disable source packet routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0 
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0

# Ignore send redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Block SYN attacks
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 5

# Log Martians
net.ipv4.conf.all.log_martians = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0 
net.ipv6.conf.default.accept_redirects = 0

# Ignore Directed pings
net.ipv4.icmp_echo_ignore_all = 1

Load the new rules:

service procps start

PHP hardening

If you’re using PHP, these are changes worth making in /etc/php5/apache2/php.ini in order to improve the security of PHP:

  1. Add exec, system, shell_exec, and passthru to disable_functions.
  2. Change expose_php to Off.
  3. Ensure that display_errors, track_errors and html_errors are set to Off.

Apache hardening

If you’re using Apache web server, it’s worth making sure you have the following parameters set in /etc/apache2/conf-enabled/security.conf to make sure Apache is suitably hardened:

ServerTokens Prod
ServerSignature Off
TraceEnable Off
Header unset ETag
FileETag None

For these to take effect you’ll need to enable mod_headers:

ln -s /etc/apache2/mods-available/headers.load /etc/apache2/mods-enabled/headers.load

Then restart Apache:

service apache2 restart

Install and configure ModSecurity

If you’re using Apache, the web application firewall ModSecurity is a great way to harden your web server so that it’s much less vulnerable to probes and attacks. Firstly, install the necessary packages:

apt-get install libapache2-mod-security2

Prepare to enable the recommended configuration:

mv /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

Then edit /etc/modsecurity/modsecurity.conf:

  1. Set SecRuleEngine to On to activate the rules.
  2. Change SecRequestBodyLimit and SecRequestBodyInMemoryLimit to 16384000 (or higher as needed) to increase the file upload size limit to 16 MB.

Next, install the Open Web Application Security Project Core Rule Set:

cd /tmp
wget https://github.com/SpiderLabs/owasp-modsecurity-crs/archive/master.zip
apt-get install zip
unzip master.zip
cp -r owasp-modsecurity-crs-master/* /etc/modsecurity/
mv /etc/modsecurity/modsecurity_crs_10_setup.conf.example /etc/modsecurity/modsecurity_crs_10_setup.conf
ls /etc/modsecurity/base_rules | xargs -I {} ln -s /etc/modsecurity/base_rules/{} /etc/modsecurity/activated_rules/{}
ls /etc/modsecurity/optional_rules | xargs -I {} ln -s /etc/modsecurity/optional_rules/{} /etc/modsecurity/activated_rules/{}

To add the rules to Apache, edit /etc/apache2/mods-available/security2.conf and add the following line near the end, just before </IfModule>:

Include "/etc/modsecurity/activated_rules/*.conf"

Restart Apache to active the new security rules:

service apache2 restart

Install and configure mod_evasive

If you’re using Apache then it’s a good idea to install mod_evasive to help protect against denial of service attacks. Firstly install the package:

apt-get install libapache2-mod-evasive

Next, set up the log directory:

mkdir /var/log/mod_evasive
chown www-data:www-data /var/log/mod_evasive

Configure it by editing /etc/apache2/mods-available/evasive.conf:

  1. Uncomment all the lines except DOSSystemCommand.
  2. Change DOSEmailNotify to your email address.

Link the configuration to make it active in Apache:

ln -s /etc/apache2/mods-available/evasive.conf /etc/apache2/mods-enabled/evasive.conf

Then activate it by restarting Apache:

service apache2 restart

Install and configure rootkit checkers

It’s highly desirable to get alerted if any rootkits are found on your server, so let’s install a couple of rootkit checkers:

apt-get install rkhunter chkrootkit

Next, let’s make them do something useful:

  1. In /etc/chkrootkit.conf, change RUN_DAILY to "true" so that it runs regularly, and change "-q" to "" otherwise the output doesn’t make much sense.
  2. In /etc/default/rkhunter, change CRON_DAILY_RUN and CRON_DB_UPDATE to "true" so it runs regularly.

Finally, let’s run these checkers weekly instead of daily, because daily is too annoying:

mv /etc/cron.weekly/rkhunter /etc/cron.weekly/rkhunter_update
mv /etc/cron.daily/rkhunter /etc/cron.weekly/rkhunter_run
mv /etc/cron.daily/chkrootkit /etc/cron.weekly/

Install Logwatch

Logwatch is a great tool which provides regular reports nicely summarising what’s been going on in the server logs. Install it like this:

apt-get install logwatch

Make it run weekly instead of daily, otherwise it gets too annoying:

mv /etc/cron.daily/00logwatch /etc/cron.weekly/

Make it show output from the last week by editing /etc/cron.weekly/00logwatch and adding --range 'between -7 days and -1 days' to the end of the /usr/sbin/logwatch command.

Enable automatic security updates

N.B. Be warned that enabling automatic updates can be potentially dangerous for a production server in a live environment. Only enable this for a server in such an environment if you really know what you are doing.

Run this command:

dpkg-reconfigure -plow unattended-upgrades

Then choose Yes.

Enable process accounting

Linux process accounting keeps track of all sorts of details about which commands have been run on the server, who ran them, when, etc. It’s a very sensible thing to enable on a server where security is a priority, so let’s install it:

apt-get install acct
touch /var/log/wtmp

To show users’ connect times, run ac. To show information about commands previously run by users, run sa. To see the last commands run, run lastcomm. Those are a few commands to give you an idea of what’s possible; just read the manpages to get more details if you need to.

Edit: I recently threw together a quick Bash script to send a weekly email with a summary of user activity, login information and commands run. To get the same report yourself, create a file called /etc/cron.weekly/pacct-report containing the following (don’t forget to make this file executable) (you can grab this from GitHub if you prefer):

#!/bin/bash

echo "USERS' CONNECT TIMES"
echo ""

ac -d -p

echo ""
echo "COMMANDS BY USER"
echo ""

users=$(cat /etc/passwd | awk -F ':' '{print $1}' | sort)

for user in $users ; do
  comm=$(lastcomm --user $user | awk '{print $1}' | sort | uniq -c | sort -nr)
  if [ "$comm" ] ; then
    echo "$user:"
    echo "$comm"
  fi
done

echo ""
echo "COMMANDS BY FREQUENCY OF EXECUTION"
echo ""

sa | awk '{print $1, $6}' | sort -n | head -n -1 | sort -nr

Things I haven’t covered

There are some additional issues you might want to consider which I haven’t covered here for various reasons:

  1. This guide assumes your Ubuntu server is on a network behind a firewall of some kind, whether that’s a hardware firewall of your own, EC2 security rules on Amazon Web Services, or whatever; and that the firewall is properly configured to only allow through the necessary traffic. However, if that’s not the case then you’ll need to install and configure a firewall on the Ubuntu server itself. The recommended software for this on Ubuntu is ufw.
  2. If you’re running an SSH server then you’re often told that you must install a tool such as fail2ban immediately if you don’t want your server to be hacked to death within seconds. However, I’ve maintained servers with publicly-accessible SSH servers for many years, and I’ve found that simply moving SSH to a different port solves this problem far more elegantly. I monitor logs in order to identify incoming hacking attempts, and I haven’t seen a single one in the many years I’ve been doing this. However, using this “security by obscurity” method doesn’t mean that such an attack can’t happen, and if you don’t watch your logs regularly and respond quickly to them as I do, then you would be well advised to install fail2ban or similar as a precaution, in addition to moving your SSH server to another port as described above.
  3. Once you’ve hardened your server, you’re advised to run some vulnerability scans and penetration tests against it in order to check that it’s actually as invincible as you’re now hoping it is. This is a topic which requires a post all of its own so I won’t be covering it in any detail here, but a good starting point if you’re not already familiar with it is the excellent Nmap security scanner.
  • Hi,

    I use Ubuntu 14.04 x64 and I followed your tutorial to install mod_security on my server:

    sudo apt-get install libapache2-mod-security2
    sudo mv /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf
    sudo nano /etc/modsecurity/modsecurity.conf

    Make the following options so:
    SecRuleEngine On
    SecRequestBodyLimit 50000000
    SecRequestBodyInMemoryLimit 50000000

    cd /tmp
    sudo wget https://github.com/SpiderLabs/owasp-modsecurity-crs/archive/master.zip
    sudo apt-get install zip
    sudo unzip master.zip
    sudo cp -r owasp-modsecurity-crs-master/* /etc/modsecurity/

    sudo mv /etc/modsecurity/modsecurity_crs_10_setup.conf.example /etc/modsecurity/modsecurity_crs_10_setup.conf

    sudo ls /etc/modsecurity/base_rules | xargs -I {} sudo ln -s /etc/modsecurity/base_rules/{} /etc/modsecurity/activated_rules/{}

    sudo
    ls /etc/modsecurity/optional_rules | xargs -I {} sudo ln -s
    /etc/modsecurity/optional_rules/{} /etc/modsecurity/activated_rules/{}

    sudo nano /etc/apache2/mods-available/security2.conf
    add the following line near the end, just before :
    Include “/etc/modsecurity/activated_rules/*.conf”

    sudo service apache2 restart
    sudo rm -rfv /tmp/*

    sudo apt-get install libapache2-mod-evasive
    sudo mkdir /var/log/mod_evasive
    sudo chown www-data:www-data /var/log/mod_evasive
    sudo nano /etc/apache2/mods-available/evasive.conf

    Make it so:

    DOSHashTableSize 3097
    DOSPageCount 2
    DOSSiteCount 50
    DOSPageInterval 1
    DOSSiteInterval 1
    DOSBlockingPeriod 10

    DOSEmailNotify my_emailaddress@gmail.com

    DOSSystemCommand "su - someuser -c '/sbin/... %s ...'"

    DOSLogDir "/var/log/mod_evasive"

    sudo ln -s /etc/apache2/mods-available/evasive.conf /etc/apache2/mods-enabled/evasive.conf
    sudo service apache2 restart
    sudo apt-get install rkhunter chkrootkit

    sudo nano etc/chkrootkit.conf
    RUN_DAILY=”true”
    RUN_DAILY_OPTS=””

    sudo nano /etc/default/rkhunter
    CRON_DAILY_RUN=”true”
    CRON_DB_UPDATE=”true”

    sudo mv /etc/cron.weekly/rkhunter /etc/cron.weekly/rkhunter_update
    sudo mv /etc/cron.daily/rkhunter /etc/cron.weekly/rkhunter_run
    sudo mv /etc/cron.daily/chkrootkit /etc/cron.weekly/

    sudo apt-get install logwatch
    sudo mv /etc/cron.daily/00logwatch /etc/cron.weekly/
    sudo nano /etc/cron.weekly/00logwatch
    /usr/sbin/logwatch –output mail –range ‘between -7 days and -1 days’

    sudo apt-get install acct
    sudo touch /var/log/wtmp

    sudo a2enmod modsecurity
    sudo /etc/init.d/apache2 force-reload

    However when I execute: sudo a2enmod modsecurity

    lupocatttivo@octane:~$ sudo a2enmod modsecurity
    ERROR: Module modsecurity does not exist!

    I even tried this:
    lupocatttivo@octane:~$ sudo apt-get install libapache2-mod-security2
    Reading package lists… Done
    Building dependency tree
    Reading state information… Done
    libapache2-mod-security2 is already the newest version.
    0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

    Any help will be appreciated.

    Thanks in advance.

    • For the a2enmod command, I think it needs a hyphen, i.e. sudo a2enmod mod-security.

      Beyond that I’m not sure what your problem is exactly?

      • great post. But has some problems.
        wrong lines: (archives doesn’t exist)
        etc/modsecurity/modsecurity.conf-recommended
        mv /etc/modsecurity/modsecurity_crs_10_setup.conf.example
        and
        /etc/modsecurity/activated_rules/{}
        error example
        ln: Failed to create symbolic link “/etc/modsecurity/activated_rules/modsecurity_crs_55_marketing.conf ‘File already exists
        etc, etc

        • I’m sorry you’re experiencing problems, but I can’t really tell anything from that I’m afraid. I know that plenty of people have worked through this procedure fine, so I think it’s more likely to be something odd with your setup.

          • thanks. i’m trying again. But is /etc/apache2/mods-available/mod-evasive.conf or /etc/apache2/mods-available/evasive.conf ???
            and in the “.conf” is: DOSLogDir /var/log/mod_evasive or DOSLogDir “/var/log/apache2/mod_evasive.log” ??
            thanks again

    • anomalous3

      I had to hunt this down; use “$sudo a2enmod security2”

    • maykel franco

      Thanks for all, good guide hardering!!!

  • Thanks for this post, it’s a nice follow-up to the fanclub guide.

  • Leo

    Hi, great post, I have a problem with the phpmyadmin the mod_security block the phpmyadmin, how can solve the problem?

    Thanks

    • I don’t use phpMyAdmin so I’m afraid I’m not familiar with it, but googling “phpmyadmin mod_security” turns up various possibilities, so I’d suggest giving those a try and seeing how you get on.

      • Leo

        Ok thanks

  • Nico Harder

    If installed all your security stuff and now i can’t connect by Webbrowser

    Forbidden

    You don’t have permission to access /index.php
    on this server.

    it seems to be a problem with modsecurity if comment it out in “/etc/apache2/mods-available/security2.conf” the server works

    • In which case it will probably be one of the ModSecurity rules clashing with your PHP application somehow. You’d need to go through the Apache logs in /var/log/apache to look for clues. Debugging ModSecurity can be quite tricky and time-consuming…

    • No Name

      Good! Now your server is secure. Unplug RJ45 too!

    • Jacob Kincaid

      I added this line to modsecurity.conf in /etc/modsecurity
      SecRuleRemoveById 970903

      This prevents it from checking for code leakage on outbound traffic, since the code isn’t sensitive that rule can safely be disabled.
      Before that I also had to tell the server to accept requests using an IP address.

      Like Matt Brock said, check the logs!

      • daveman

        “Before that I also had to tell the server to accept requests using an IP address.”
        Please how?

        • Jacob Kincaid

          It is another rule that needs to be blocked by Id number, with SecRuleRemoveById xxxxxx, if you check the logs it will tell you what the Id number is, don’t forget to use tools like grep and tail to make log checking easier

          • daveman

            Amazing! It’s working. I also made separately custom rule conf, where I put rules with id and then symlinked it into active_rules.
            Thank you very much

          • BIYOHNE Fabrice

            hello dear daveman, please can show me how you solve the problem with modsecurity? i have error accessing webserver after modsecuty installation and configuration. i know i have to add something like SecRuleRemoveById XXXXXX. but how to do that?

            thx in advance for the support

  • Mezbaur Rahman

    brilliant post Matt.

  • Arshad Vayani

    Thanks a lot for sharing such valuable information. I have a learned a lot about security hardening through your guide.

    A quick question: How to receive process accounting report via email through bash script you have shared in the end?

    • The report will be sent automatically via email each week. You just need to make sure that outgoing email is set up properly on your server.

      • Pankaj Gupta

        Thanks Matt but where we have to mention email id for weekly report ?

        • This email will get sent automatically if outgoing email is set up properly on your server. Setting up email is a fairly complex task outside the scope of this article, and it’s not something I can realistically summarise in a comment.

  • rat

    awesome! thank you for puting all this in one spot. I would like to add a comment about fail2ban … maybe you can edit it in some where: (further details here https://www.digitalocean.com/community/tutorials/how-to-install-and-use-fail2ban-on-ubuntu-14-04 )

    apt-get install fail2ban
    sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

    edit the things you like in /etc/fail2ban/jail.local

  • Shahar Galukman

    Great post, surely looks like great solutions for the nasty hacks out there.

  • Obaid Ullah Khan

    Gr8 post Matt

  • Obaid Ullah Khan

    Thanks a lot …

  • Infusion

    Hi Matt. I’ve used the tips above to setup a a few webservers now, but I always seem to run into the same issue. The php mailer function seems to fail and I’ve had to setup smtp to run in its place.
    The PHP hardening section of this post doesn’t seem to reference anything to do with the mailer.
    Any ideas ?

    • Hard to say for sure without seeing some logs/debug output/etc., but I’d guess it will probably be one of the ModSecurity rules clashing with the PHP mailer. As I mentioned in another comment, you’d need to go through the Apache logs in /var/log/apache to look for clues, and debugging ModSecurity can be quite tricky and time-consuming.

  • great post.

  • With all these logs and monitors, is there an elegant solution to having say a terminal that auto-updates the logs being viewed on a remote computer?

    For example, I’d like to monitor apache and other logs in real-time or with a 5-10 minute refresh time in Terminal on the Mac. I’m a newbie to sysadmin and I have heard of corncobs etc but I’m not entirely sure how I would go about this. I constantly hear stories of “ninja sys admins” sitting with multiple monitors and at least one is a constant stream of live server data, I would like a set up like that for learning and training on my VPS.
    Any suggestions would be greatly appreciated 🙂

    • Why would you not simply run tail?

      • Yeah tail is your best option, the above is the most elegant solution, I follow all Matt Brock’s steps and he has never let me down, plus if he did he got us this far, can’t blame him for anything.

  • Wilko de Vries

    I wish to express my gratitude for creating this website & blog.
    I’m new to Linux Ubuntu Server, and this helps me a lot to take it all in.
    Keep up the great work, and i wish you the best luck as a freelancer ( i don’t think you need it, but anyway 😉

  • installing zip doesn’t automatically add unzip. For those who run into issues, check to see if you missed the error “missing unzip”. “apt-get install unzip” will fix it.

    Could you shed some light on adding a modsecurity whitelist? Other sources point to creating a whitelist.conf file and adding exceptions there which seems elegant. However I can’t get my exceptions to be honoured…

    • Whenever I’ve installed zip, unzip has been installed automatically. No harm in double-checking this though, for those having problems.

      Adding some info about a whitelist sounds like a good suggestion. I’ll see if I can look into that at some point.

  • sin

    I really appreciate your blog and this post specially. I’d like to contribute adding the firewall configuration that I use (ufw):

    sudo ufw default deny (this set firewall to block everything)
    sudo ufw allow from YOUR_IP to any port 22

    open all ports that you need (ex: sudo ufw allow 80 or sudo ufw allow 8080/tcp etc etc)

    sudo ufw enable

    Hope it helps
    Edit: you can also use hosts.deny and hosts.allow to add an extra security layer. For example (for ssh):
    set in hosts.deny –> sshd: ALL
    set in hosts.allow –> sshd: YOUR_IP

    After reading all the post, I realized it’s the point 1 of your uncovered things 🙂

  • Boaty Mcboatface

    Hi there just wanted to let you know I found this guide very useful, thank you for the good read.

  • sai arun jana

    Hi matt,
    while loading procps, I’m getting like procps stop/waiting ..

    • A quick google suggests this can be caused by a bug in the package. Try upgrading your Ubuntu installation.

  • cyberfarer

    I don’t see how the script sends an email so …

    !/bin/bash

    OUTFILE=mktemp || exit 1
    EMAIL=me@gmail.com

    echo "USERS' CONNECT TIMES" >> $OUTFILE
    echo "" >> $OUTFILE

    /usr/bin/ac -d -p &gt;&gt; $OUTFILE

    echo "" &gt;&gt; $OUTFILE
    echo "COMMANDS BY USER" &gt;&gt; $OUTFILE
    echo "" &gt;&gt; $OUTFILE

    users=$(cat /etc/passwd | awk -F ':' '{print $1}' | sort)

    for user in $users ; do
    comm=$(lastcomm --user $user | awk '{print $1}' | sort | uniq -c | sort -nr)
    if [ "$comm" ] ; then
    echo "$user:" &gt;&gt; $OUTFILE
    echo "$comm" &gt;&gt; $OUTFILE
    fi
    done

    echo "" &gt;&gt; $OUTFILE
    echo "COMMANDS BY FREQUENCY OF EXECUTION" &gt;&gt; $OUTFILE
    echo "" &gt;&gt; $OUTFILE

    sa | awk '{print $1, $6}' | sort -n | head -n -1 | sort -nr &gt;&gt; $OUTFILE

    if [ -s “$OUTFILE” -a -n “$EMAIL” ]; then
    (
    echo “Subject: $(hostname -f) – Pacct Weekly Report”
    echo “To: $EMAIL”
    echo “”
    cat $OUTFILE
    ) | /usr/sbin/sendmail $EMAIL
    fi

    rm -f $OUTFILE
    exit 0

    • The output from the cronjob goes to root by default, so if you’ve got root email going to yourself (which you should have) then you’ll get the output automatically. But you can do it this way if you prefer.

  • Pingback: Ubuntu Server – Links and Websites Used For Server Installation Project – Reach Into My Brain()

  • Abu

    After successfully finishing all steps, when i login into the phpmyadmin and import one databse it shows “You don’t have permission to access /phpmyadmin/import.php on this server”. Please suggest.. How to solve this problem?