After some research I ended up using the excellent Initializr to produce a template website built from HTML5 Boilerplate (H5BP) and Twitter Bootstrap. H5BP provides a solid HTML5 core with optimisations for JavaScript, CSS, Apache, Google Analytics, etc., and Twitter Bootstrap provides a simple but very powerful HTML/CSS layout framework which cleverly handles things like rendering on mobile devices so that you don’t have to worry about it.

It was very easy to modify the template and build the simple marketing website I wanted. The end result isn’t the most advanced design ever created, and it’s instantly recognisable as a Bootstrap site to anyone else who’s used it, but that’s OK. I’m just trying to market my business, not win design awards. I’d certainly recommend Initializr to anyone else who understands HTML and CSS and wants to get a decent site up and running as quickly as possible.

The end result can be seen at mattbrock.co.uk.

(If you’re interested in hiring me for sysadmin consultancy work, just email me.)

ImageMagick installed (“brew install imagemagick” if you’re on a Mac with Homebrew installed, which it really ought to be). You’ll also need to make sure you have base64 installed, which it probably already will be. If you’re copying and pasting this script, make sure you replace the ^M near the bottom of the script with a literal ^M (normally you type Ctrl-V then Ctrl-M to get a literal ^M). The script will take the vCard file you specify as input and create a subdirectory called photos containing the photos from your contacts.

#!/bin/bash

progname=$(basename $0)
usage="Usage:\t$progname -f vCard_file \n\t$progname -h"

while getopts "f:h" options ; do
  case $options in
    f) vcard_file=$OPTARG ;;
    h) echo -e $usage ; exit ;;
  esac
done

if [ ! "$vcard_file" ] ; then
  echo -e $usage
  exit 1
fi

if [[ ! $(head -1 $vcard_file) =~ "BEGIN:VCARD" ]] ; then
  echo "$vcard_file does not appear to be a vCard file"
  exit 1
fi

if ! which base64 > /dev/null ; then
  echo "base64 is not installed"
  exit 1
fi

if ! which convert > /dev/null ; then
  echo "ImageMagick is not installed"
  echo "Either install it or comment out the ImageMagick stuff"
  exit 1
fi

if [ -d photos ] ; then
  echo "photos directory already exists"
  exit 1
fi

if ! mkdir photos ; then
  echo "Failed to create photos directory"
  exit 1
fi

regex1="^N:"
regex2="^PHOTO;"
regex3=":|;"

echo -ne "Extracting photos"
cat $vcard_file | while read line ; do
  if [[ $line =~ $regex1 ]] ; then
    filename=$(echo $line | awk -F '[:;]' '{printf("%s%s",$3,$2)}' | \
      sed 's/[^a-zA-Z]//g')
    echo -ne "."
    cat /dev/null > photos/$filename.tmp
  elif [[ $line =~ $regex2 ]] ; then
    echo $line | sed 's/PHOTO;ENCODING=b;TYPE=JPEG://' >> photos/$filename.tmp
  elif [[ ! $line =~ $regex3 ]] ; then
    echo $line >> photos/$filename.tmp
  fi
done
echo -ne " done\n"

echo -ne "Removing empty photos..."
find photos -empty -exec rm -f {} +
echo -ne " done\n"

echo -ne "Converting photos"
cd photos
for file in *.tmp ; do
  filename=$(echo $file | awk -F '.' '{print $1}')
  echo -ne "."
  cat $file | tr -d '\n' | tr -d '^M' | base64 -d > $filename.jpg
  convert $filename.jpg -resize 201x201 $filename.jpg
  rm -f $file
done
echo -ne " done\n"

ngrep and watch are installed. Then run the following Bash script (replacing eth0 with the correct interface if it’s not eth0):

#!/bin/bash

ngrep -il -d eth0 -W byline "x-forwarded-for" "port 80" | grep -i \
  x-forwarded-for | awk -F '[., ]' '{printf( "%s.%s.%s.%s\n", $2,$3,$5,$5 );}' \
  > /tmp/ngrep.tmp &
watch -tn 10 'cat /tmp/ngrep.tmp | sort -n | uniq -c | sort -nr | head -30'

When you run this script you should see a continuously updating display which looks something like this:

80 11.22.33.44
60 55.66.77.88
40 44.33.22.11
20 88.77.66.55
...

It may start to slow down after a while. If that happens then just hit CTRL-C and run it again.

#!/usr/local/bin/python
# -*- coding: utf-8 -*-

import re
from optparse import OptionParser

parser = OptionParser()
parser.add_option("-f", "--file", dest="filename", metavar="FILE", 
  help="read reminders from ICS FILE")
parser.add_option("-c", "--include-completed", action="store_true", 
  dest="include_completed", help="include completed tasks")
(options, args) = parser.parse_args()

if not options.filename:
  print "No filename specified; -h for help"
  exit(1)

with open(options.filename) as file:
  ics_export = file.readlines()
file.close

todos = []
for ics_line in ics_export:
  if "STATUS:COMPLETED" in ics_line and options.include_completed is None:
    exclude = True
  elif "STATUS:" in ics_line:
    exclude = False
  if "SUMMARY:" in ics_line and exclude is False:
    bits = re.split(":", ics_line)
    todos.append(bits[1].rstrip())

todos.sort(key=lambda y: y.lower())
for todo in todos:
  print "•", todo

^{* Synced to my iPhone and other Macs via ownCloud on my Linux server, but that’s a story for another day.}

Installation and initial configuration

Install the required packages and prepare the configuration file:

yum install pacemaker
cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf

Change bindnetaddr to your network address (e.g. 192.168.1.0) in /etc/corosync/corosync.conf.

Create /etc/corosync/service.d/pcmk and put the Pacemaker startup config into it:

service {
        name: pacemaker
        ver: 1 
}

Start corosync and pacemaker:

service corosync start
service pacemaker start

Make sure they start on boot:

chkconfig corosync on
chkconfig pacemaker on

Repeat all of the above on the second node.

Then disable STONITH on the primary node:

crm configure property stonith-enabled=false

Add a floating IP on the primary node (change IP address and netmask as needed):

crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 params ip=192.168.1.110
  cidr_netmask=24 op monitor interval=30s

Disable quorum on the primary node (necessary for running only two nodes):

crm configure property no-quorum-policy=ignore

Prevent resources from moving back after a node recovers:

crm configure rsc_defaults resource-stickiness=100

Use the commands in the Administration section below to ensure that the floating IP is on the correct machine – if this is not the case then temporarily shut down Pacemaker on whichever machine has the floating IP so that it moves across to the desired node:

service pacemaker stop

Administration

Display the current configuration:

crm configure show

Status monitor:

crm status

Show which node has the cluster IP:

crm resource status ClusterIP

Reset the config to default:

cibadmin -E --force

Verify that there are no problems with the current config:

crm_verify -L

This problem turned out to be caused by the HAProxy load balancer not adding HttpOnly to its persistence (sticky session) cookies, so I checked the HAProxy manual and did some googling, but I couldn’t find any way of making HAProxy’s persistence cookies add the HttpOnly flag.

The HAProxy website says “if you want to suggest a useful feature, please discuss it on the mailing list” so I’ve tried to do just that but have not had any response yet. In the meantime, I’ve modified the code myself so that the HttpOnly flag is added to all persistence cookies. For version 1.4.21 of HAProxy, patching src/proto_http.c with the following will add my modification to achieve this:

5348a5349,5350
> 			len += sprintf(trash+len, "; HttpOnly");
> 

Hopefully this is something the HAProxy developers will add permanently as an option, otherwise it’s quite awkward for HAProxy users needing to pass compliance tests whilst using persistence cookies.

Edit: Willy, the developer of HAProxy, has replied on the mailing list to indicate that he’ll be adding an ‘httponly’ option to the ‘cookie’ parameter in version 1.5 of HAProxy, so that’s great news. In the meantime, my little hack above can be used for those needing HttpOnly on version 1.4.

Edit: This has now been added as an option in version 1.4.22 of HAProxy.

I’m using LVM for the disk volumes because that’s supposedly better for IO efficiency than using disk images stored on the host filesystem, so a bit of familiarity with LVM is ideally needed before getting started.

Initial host setup

Firstly it’s necessary to make sure you have all the necessary software installed:

yum -y groupinstall Virtualization "Virtualization Client"
  "Virtualization Platform" "Virtualization Tools" ;
  yum -y install libguestfs-tools

Next check that libvirtd is running:

service libvirtd status

If not, make sure that messagebus and avahi-daemon are running, then start libvirtd:

service messagebus start
service avahi-daemon start
service libvirtd start

Use chkconfig to ensure that all three of these services start automatically on system boot.

Next it’s necessary to set up the network bridge so that the virtual machines can function on the network in the same way as physical servers. To do this, copy /etc/sysconfig/network-scripts/ifcfg-eth0 (or whichever is the file for the active network interface) to /etc/sysconfig/network-scripts/ifcfg-br0.

In /etc/sysconfig/network-scripts/ifcfg-eth0, comment out all the lines for ‘BOOTPROTO’, ‘DNS1′, ‘GATEWAY’, ‘IPADDR’ and ‘NETMASK’, then add this line:

BRIDGE="br0"

Then edit /etc/sysconfig/network-scripts/ifcfg-br0, comment out the ‘HWADDR’ line, change the ‘DEVICE’ to “br0″, and change the ‘TYPE’ to “Bridge” (or add this line in if it’s not here; note the capital “B” at the start of “Bridge”).

Then restart the network:

service network restart

The bridge should now be up and running. You can check its status with:

ifconfig
brctl show

Creating the disk volumes for a new virtual machine

We need to create new LVM volumes for the root and swap partitions in the new virtual machine. I’m assuming LVM is already being used, that the volume group is called ‘sysvg’, and that there is sufficient free space available in the sysvg group for the new volumes. If your volume group has a different name then just modify the instructions below accordingly. Change the volume sizes to suit your requirements:

lvcreate -L 20G -n vm-root sysvg
lvcreate -L 4096M -n vm-swap sysvg

Installing the operating system on the new virtual machine

Here I’m installing CentOS 6 on the guest machine using Kickstart, although I will also explain how to perform a normal non-automated installation. You’ll need to modify the instructions accordingly to install different operating systems.

To make CentOS easily available for the installation, firstly make sure you have Apache installed and running. If necessary, install it with:

yum -y install httpd

Then start it with:

service httpd start

Then create the directory /var/www/html/CentOS and copy the contents of the CentOS DVDs into it.

If you’re using Kickstart then you’ll need these lines in your Kickstart config file to make sure that it can find the files from the CentOS DVDs. The IP address of the host in this example is 192.168.1.1, so change that as needed:

install
url --url http://192.168.1.1/CentOS

These lines are also required to make sure that Kickstart can find and use the LVM volumes created earlier:

zerombr
clearpart --all --initlabel
bootloader --location=mbr
part / --fstype ext4 --size 1 --grow --ondrive=vda
part swap --size 1 --grow --ondrive=vdb

Once the Kickstart file is ready, call it ks.cfg and copy it to /var/www/html

This command installs CentOS on the guest using a Kickstart automated installation. The guest is called ‘vm’, it has a dedicated physical CPU core (core number 2) and 1 GB of RAM allocated to it. Again, the IP address of the host is 192.168.1.1, so change that as needed:

virt-install --name=vm --cpuset=2 --ram=1024
  --network bridge=br0 --disk=/dev/mapper/sysvg-vm--root
  --disk=/dev/mapper/sysvg-vm--swap --vnc --vnclisten=0.0.0.0
  --noautoconsole --location /var/www/html/CentOS
  --extra-args "ks=http://192.168.1.1/ks.cfg"

The installation screen can be seen by connecting to the host via VNC. This isn’t necessary for a Kickstart installation (unless something goes wrong). If you want to do a normal install rather than a Kickstart install then you will need to use VNC to get to the installation screen, and in that case you’ll want to use the virt-install command above but just leave off everything from ‘–extra-args’ onwards.

Also, you may want to install directly from a CDROM image, in which case replace the ‘–location’ bit with ‘–cdrom=’ and the path to the CD image, e.g. to install Ubuntu in your VM you might put ‘–cdrom=/tmp/ubuntu-12.04.1-server-i386.iso’.

(If virtual servers are already using VNC on the host then you will need to add the appropriate number to the VNC port number to connect to, e.g. the standard VNC port is 5900, and if there are already two virtual servers using VNC on the host then you will need to connect VNC to port 5902 for this install.)

General administration of virtual machines

Once you’ve got your virtual machine installed, you’ll need to know the various commands for everyday administration of KVM virtual machines. In these examples, change the name of the VM from ‘vm’ to whatever yours is called.

To show general info about virtual machines, including names and current state:

virsh list --all

To see a top-style monitor window for all VMs:

virt-top

To show info about a specific virtual machine:

virsh dominfo vm

To start a virtual machine:

virsh start vm

To pause a virtual machine:

virsh suspend vm

To resume a virtual machine:

virsh resume vm

To shut down a virtual machine (the ‘acpid’ service must be running on the guest for this to work):

virsh shutdown vm

To force a hard shutdown of a virtual machine:

virsh destroy vm

To remove a domain (don’t do this unless you’re sure you really don’t want this virtual machine any more):

virsh undefine vm

Cloning virtual machines

To clone a guest VM, firstly it’s necessary to create new disk volumes for the clone, then we use the virt-clone command to clone the existing VM:

lvcreate -L 20G -n newvm-root sysvg
lvcreate -L 4096M -n newvm-swap sysvg
virt-clone -o vm -n newvm -f /dev/mapper/sysvg-newvm--root
  -f /dev/mapper/sysvg-newvm--swap

Then dump the XML for the new VM:

virsh dumpxml newvm > /tmp/newvm.xml

Edit /tmp/newvm.xml. Look for the ‘vcpu’ line and change the ‘cpuset’ number to the CPU core you want to dedicate to this VM. Then make this change effective:

virsh define /tmp/newvm.xml

You’ll also need to grab the MAC address from the XML. Keep this available as we’ll need it in a minute:

grep "mac address" /tmp/newvm.xml | awk -F ' '{print $2}'

Start up the new VM and connect to it via VNC as per the instructions in the Installation section above. Edit /etc/sysconfig/network and change the hostname to whatever you want to use for this new machine. Then edit /etc/sysconfig/network-scripts/ifcfg-eth0 and change the ‘HOSTNAME’ and ‘IPADDR’ to the settings you want for this new machine. Change the ‘HWADDR’ to the MAC address you obtained a moment ago, making sure that the letters are capitalised.

Then reboot and the new VM should be ready.

Backing up and migrating virtual machines

In order to take backups and to be able to move disk volumes from virtual machines to other hosts, we basically need to create disk image files from the LVM volumes. We’ll first snapshot the LVM volume and take the disk image from the snapshot, as this significantly reduces the amount of time that the VM needs to remain paused (i.e. effectively offline) for. We remove the snapshot at the end of the process so that the VM’s IO is not negatively affected.

This disk image, once created, can then be stored in a separate location as a backup, and/or transferred to another host server in order to copy or move the VM there.

So, make sure that the VM is paused or shut down, then create a LVM snapshot, then resume the VM, then create the image from the snapshot, then remove the snapshot:

virsh suspend vm
lvcreate -L 100M -n vm-root-snapshot -s /dev/sysvg/vm-root
virsh resume vm
dd if=/dev/mapper/sysvg-vm--root--snapshot
  of=/tmp/vm-root.img bs=1M
lvremove /dev/mapper/sysvg-vm--root--snapshot

You can then do what you like with /tmp/vm-root.img – store it as a backup, move it to another server, and so forth.

In order to restore from it or create a VM from it on a new server, firstly use ‘lvcreate’ to create the LVM volume for restore if it isn’t already there, then copy the disk image to the LVM volume:

dd if=/tmp/vm-root.img of=/dev/mapper/sysvg-vm--root bs=1M

You may also need to perform this procedure for the swap partition depending on what you are trying to achieve.

You’ll also want to back up the current domain configuration for the virtual machine:

virsh dumpxml vm > /tmp/vm.xml

Then just store the XML file alongside the disk image(s) you’ve taken.

If you’re moving the virtual machine to a new server then once you’ve got the root and swap LVM volumes in place, you’ll need to create the domain for the virtual machine on the new server. Firstly edit the XML file and change the locations of disk volumes to the layout on the new server if it’s different to the old server, then define the new domain:

virsh define /tmp/vm.xml

You should then be able to start up the ‘vm’ virtual machine on the new server.

Resizing partitions on a guest

Let’s say we want to expand the root partition on our VM from 20G to 25G. Firstly make sure the VM is shut down, then use virt-filesystems to get the information we need for the resize procedure:

virsh shutdown vm
virt-filesystems -lh -a /dev/mapper/sysvg-vm--root

This will probably tell you that the available filesystem on that volume is /dev/sda1, which is how these tools see the virtual machine’s /dev/vda1 partition. We’ll proceed on the basis that this is the case, but if the filesystem device name is different then alter the command below accordingly.

Next we create a new volume, then we perform the virt-resize command from the old volume to the new volume, then we set the new volume as the active root partition for our domain:

lvcreate -L 25G -n vm-rootnew sysvg
virt-resize --expand /dev/sda1 /dev/mapper/sysvg-vm--root
  /dev/mapper/sysvg-vm--rootnew
lvrename /dev/sysvg/vm-root /dev/sysvg/vm-rootold
lvrename /dev/sysvg/vm-rootnew /dev/sysvg/vm-root
virsh start vm

Then, when you’re sure the guest is running OK with the new resized partition, remove the old root partition volume:

lvremove /dev/mapper/sysvg-vm--rootold

In PostgreSQL, sorts larger than a certain size will get performed on disk instead of in memory, and this makes them much slower as a result. Ideally all sorts should be done in memory (except for the ones that are genuinely too big to fit into your available RAM, because swapping to virtual memory should be avoided at all costs).

Seeing which sorts are happening on disk

To see which sorts are being performed on disk, the following parameters need to be enabled in /var/lib/pgsql/data/postgresql.conf:

log_destination = 'syslog'
log_temp_files = 0

PostgreSQL syslog logging goes to the ‘local0′ facility by default, so in /etc/syslog.conf make sure that you have ‘local0.none’ added to the line for /var/log/messages, and add the following line:

local0.* /var/log/pgsql

Then restart syslog and reload PostgreSQL:

service syslog restart
service postgresql reload

You should now start seeing details of your disk-based sorts in /var/log/pgsql. Each one will start with a line containing the phrase ‘LOG: temporary file’ and will show other details below this line including the SQL of the query which contained the sort. After a while you will see patterns emerging.

Making sorts happen in memory instead of on disk

If you have disk-based sorts occurring frequently, you can find the size of the largest of the frequent queries and then set this as the standard threshold for sorts. The parameter for this in /var/lib/pgsql/data/postgresql.conf is ‘work_mem’. For example, if you had frequent disk-spaced sorts using up to 64MB, you would set the following in /var/lib/pgsql/data/postgresql.conf:

work_mem = 64MB

Be very careful, however, because this setting gets used by each query in each thread, so if you set this too high you could rapidly run out of RAM and cause the system to start swapping, which would be disastrous. The best thing to do is increase this bit by bit until as many as possible of your frequent queries are sorting in RAM but without running out of available memory. To see how much RAM is actually available on your system, use the command ‘free’. As Linux tends to use up the majority of available RAM for its disk cache, the number you actually care about is the ‘free’ column in the ‘buffers/cache’ row – this is how much RAM is really available. For example, here we can see that there are just over 11 GB of free RAM available:

# free
             total       used       free     shared    buffers     cached
Mem:      12300988   12013536     287452          0     277788   10801424
-/+ buffers/cache:     934324   11366664
Swap:      4192956        240    4192716

Once you’ve dealt with sorts for frequent queries, you might find there are still the occasional disk-based sorts happening for a small number of particularly intensive queries. For those, you can increase the sort memory threshold on a per-query basis and then put it back to the standard setting once the query is finished. To do this, prepend your query with:

SET work_mem = '500MB';

changing ’500MB’ to whatever size is appropriate for you, then append the query with:

RESET work_mem;

to return it to the standard threshold. Again, be careful that you don’t use up all the system’s available RAM because then the dreaded swapping will occur.

Once you’ve done all this, as many sorts as possible will be happening in memory instead of on disk, and your PostgreSQL installation should be performing considerably better as a result of this tuning.

Getting information from the logfile into a more useful format

When PostgreSQL logs temporary files (disk-based sorts), the log entries can be rather messily distributed in the logfile, especially if you’re logging other things, such as slow queries, at the same time. I knocked up the following Bash script to quickly extract the details of the sorts into a more useful format:

#!/bin/bash

PROG=$(basename $0)
PPROG=$(echo $PROG | awk -F '.' '{print $1}')
TMPFILE1=/tmp/$PPROG.tmp1
TMPFILE2=/tmp/$PPROG.tmp2
LOGFILE=$1

if [ -z "$1" ] ; then
  echo "Usage: $PROG LOGFILE"
  exit 
fi

cat /dev/null > $TMPFILE2

grep "temporary file" $LOGFILE > $TMPFILE1

cat $TMPFILE1 | while read LINE ; do
  NO1=$(echo $LINE | awk -F '[][-]' '{print $2}' )
  NO2=$(echo $LINE | awk -F '[][-]' '{print $4}' )
  cat $LOGFILE | 
    awk -F '[][-]' "($2 ~ /$NO1/) && ($4 ~ /$NO2/) {print $0}" 
    >> $TMPFILE2
  echo "" >> $TMPFILE2
done

cat $TMPFILE2

rm -f $TMPFILE1 $TMPFILE2

Launchpad: Tedious scrolling through multiple screens of apps in order to launch one of them? No thanks. Dreadful.

Mission Control: It seems like a good idea in principle to combine Exposé and Spaces, but in doing so some useful functionality is lost: minimised windows are no longer shown; multiple desktops are now one-dimensional rather than two-dimensional; I no longer have a number in the Menu Bar telling me which desktop I’m currently on; and it’s harder to see the contents of the other desktops. All a bit of a shame really.

Full-Screen Apps: This seems initially useful, but: it doesn’t make use of multiple monitors; the keyboard shortcut only seems to work some of the time; it disables my hot corner to quickly get to the Desktop; switching between windows within a fullscreen app (e.g. Mail) is sometimes impossible; and it adds confusion because all windows already have a Zoom button, so now they all have two maximise buttons instead of one.

Resume/Auto Save/Versions: This seems like a good idea in theory, but I doubt many of the apps I use will implement this for a long time, if ever.

AirDrop: I was really looking forward to this one but didn’t realise that it only works on some recent Macs, i.e. none of mine. So that was very disappointing.

Mail: The redesign is quite nice both aesthetically and ergonomically. Nothing ground-breaking, but nice. I’ve already become very dependent on the conversation view to the point where I now wish they’d hurry up and enable it for my iPhone too.

Address Book/iCal: The horrible graphics in these are ugly and ridiculous, and Address Book has much less space for useful information as a result. I thought Apple had stopped this sort of nonsense years ago and didn’t expect to see it come back now. Also, iCal is having severe problems talking to my work calendar, and the work calendar is iCal Server running on OS X Leopard Server, so it’s particularly annoying that that doesn’t work properly [now fixed by using SSL for CalDAV]. Having said all that, I do like the new ‘planner’ pane on the ‘Day’ view – that’s now replaced ‘Week’ as my standard view as a result.

Front Row: Instead of bringing this up to date, which would have been sensible especially now that Mac minis have HDMI outputs, they’ve just removed it instead. Great. Thanks, Apple. I don’t use it that much in comparison to Plex on my Mac mini-based media centre, but it would have been nice to have functionality added instead of the whole thing being taken away. Very poor show indeed.

Terminal: I love the new ‘Blur’ option for window backgrounds, and I’ve come up with a really nice combination of blur and opacity for my active and inactive windows. But why, when I open new tabs, do they start SSHing into random servers when I don’t want them to? I presume this is something to do with Resume but it’s bloody annoying and I want it to stop! [Turns out it was necessary to change the 'New tabs open with' settings in the Startup tab in Terminal Preferences.]

UI changes: Being able to resize a window from any edge is very welcome – I wish they’d done this years ago. It’s nice to see the back of the oversized Aqua scroll bars at last (but desirable to change the scroll bar setting to show ‘Always’ in the General System Preference because this is a computer, not an iPhone). The overall look and feel is like a more subtle, streamlined version of Snow Leopard, which is quite nice generally. I worried about the icons in sidebars (in Finder and Mail most noticeably) being stupidly oversized, but thankfully I then found the ‘Sidebar icon size’ option in the General System Preference – setting that to ‘Small’ makes everything OK again. And the increased use of animation is quite enjoyable and might occasionally be vaguely useful for new computer users. All the new gesture stuff is irrelevant to me because I don’t have any of the new Macs with big trackpads, and I prefer using mice anyway (though not Apple mice because they’re an ergonomic nightmare). Oh, and suddenly finding scrolling going in the wrong direction wasn’t particularly useful, but thankfully that was easy to turn off in the Mouse System Preference.

System stuff: Make sure you have ‘Show indicator lights for open applications’ ticked in the Dock System Preference, partly because that’s a traditional behaviour which helps to actually make the Dock useful, and partly because if you don’t then Lion will randomly quit open apps without asking. Another bit of weirdness is the ‘Mail, Contacts & Calendars’ System Preference, which seems to serve no useful purpose whatsoever other than to confuse users by duplicating settings which you’ve already got in Mail, iCal, etc.

So there we go. Once you’ve turned all the annoyances off it’s basically just a neater, slicker version of Snow Leopard with a few things missing, so I’m not exactly blown away by this new OS. I think the main problem is that trying to mix Mac OS X and iOS causes a conflicting mess of too many different UI paradigms. Apple should either have stuck with tweaking the Mac OS X foundations or rebuilding the whole thing from the ground up, not ending up with something in between the two which doesn’t quite work.

Edit: It’s quite interesting to compare the rather dark tone of this review to the optimistic positivity of my Snow Leopard and Leopard reviews. It makes Lion feel like even more of a step back than a step forward.

Edit: Reading back on my reviews of previous versions of OS X, I’m reminded of what a mess Apple made of To Do Items in Mail when that was added to Leopard. I see they’ve fixed that by simply removing them again, which I guess is fair enough really.

Firstly, EC2 is much more flexible than traditional colocation. Our infrastructure can now be expanded or altered, quite radically if necessary, in hours or even minutes, just by typing a few commands or clicking a few buttons. Previously, making similar changes would have taken weeks of planning, ordering, delivery, physical installation, software installation, etc.

Secondly, EC2 works out cheaper. EC2 server instances tend to work out cheaper than their physical colocation equivalents, especially when you’re able to plan ahead with pricing by using Amazon’s ‘Reserved’ pricing instead of their standard ‘On-Demand’ pricing. Also, because of EC2′s flexibility, it’s only ever necessary to run precisely the infrastructure required at the present moment, so you’re never paying for more than you need. With a traditional colocation setup, the lack of flexibility means that you will always have more hardware than you need at the present moment in order to cope with possible traffic spikes, hardware failures, etc., and that extra hardware increases the cost even further.

Getting up and running with EC2 is pretty straightforward. You just need to provide payment details, set up security details, and then you’re free to fire up server instances and try it out. There are various instance types to choose from depending on the hardware spec you need. Much of this can now be done via the Amazon Web Services (‘AWS’) Management Console, although you can also install the EC2 command line tools in order to operate EC2 via the CLI. I prefer using the CLI, probably because I’ve been doing it that way for longer, but also I guess there are still some things you can do via the CLI which you can’t do via the AWS Management Console.

Any decent system administrator will want to set up and configure instances according to his own tastes and requirements, and the best way to do this is to start an instance from an image that’s closest to what you want, then modify it accordingly, then create your own image from that running instance. I originally started up an instance from a CentOS image provided by RightScale, then modified it heavily according to my own preferences, then created an image of my own from it. This image is what I called the ‘base’ image.

I then created several different instance types for each of the different types of servers we have in our infrastructure. For each of these I started from the ‘base’ image, installed the applications required for that instance type, then created a new image from it, so that I ended up with several images: one for web server instances, another for mail server instances, and so on. Each instance type has a certain set of events which need to occur when that instance is first run, e.g. working out its own external hostname, setting up certain directories that are needed, etc., and for this I have a startup script in /etc/rc.d/init.d on each instance which works out whether it’s a brand new instance and, if so, performs all these tasks.

Once all the required instance types have been created and made available as images which can be started as required, it’s almost possible to start up a full web application infrastructure with all the components required for it. However, there are a few extra things to think about first.

Firstly, it’s necessary to consider disaster recovery. Conveniently, most of our instances do not contain any unique data which would be missed if the instance died, and if an instance dies then it’s normally just a case of firing up a new one to take its place. Obviously requirements will vary from one company to another, but in our case only the database instance contains unique data, i.e. the live database. I solved the disaster recovery requirement for the database instance by utilising an AWS storage solution called Elastic Block Store (or ‘EBS’). An EBS volume is a storage volume which can be connected to a running instance and which can also be snapshotted. So, all the data for our live database is on an EBS volume connected to the database instance, and this gets backed up and snapshotted every night. I have a Bash script which does this automatically from cron, and which also removes any shapshots older than 28 days.

Therefore, if the database instance dies, I can just fire up a new instance and connect the EBS volume to the new instance. In the unlikely event that the EBS volume dies, I can just create a new one from the latest snapshot and connect it to the database instance. This is therefore a very handy DR solution for the live database, and EBS volumes have the added bonus of being faster than the standard instance-based storage, which is obviously ideal for a database.

All instance images and EBS snapshots are stored on S3. S3 is an exceptionally safe form of storage, so the chances of anything getting lost on it are incredibly low, and this very low level of risk might be reassurance enough for many companies. In our case, however, I’m exceptionally paranoid so I also take backups of everything we have on S3 (including all our media which is also stored there). Every month I use the rsync-style functionality within the very handy s3cmd tool to incrementally backup all our S3 buckets to a disk on a server in our office.

Another consideration is how to load balance the front-end servers. Amazon provide a great solution for this called Elastic Load Balancing. This can be used to quickly and easily create a load balancer which can be configured to distribute whatever traffic you like across whichever instances you need. It even has support for sticky sessions. The only problem I’ve found with ELB is that it requires you to point a CNAME at the load balancer rather than an A record, which means that you can’t use root domains (i.e. apple.com instead of www.apple.com) because the DNS spec does not allow a root domain to be a CNAME. There are various hacks to get around this, but none of them are perfect, and it’s a problem I hope Amazon will solve soon. In the meantime, this thread in the AWS developer forums gives more information.

It’s also necessary to consider the delivery of email. Unfortunately, due to the ease with which server infrastructures can be created on EC2, a lot of spammers use it to send email, which means that EC2 IP addresses are heavily blacklisted. For each EC2 instance you wish to send email from, therefore, you need to follow a process which helps Amazon to ensure that you’re not sending spam. Firstly, you need to assign an Elastic IP to the instance. An Elastic IP is an IP address you can request and then keep for your own use, assigning it to whichever instance you like at any given time instead of using whichever IP address that instance happened to have by default. Once the Elastic IP is assigned to the mail server instance, you just need to tell Amazon to set up reverse DNS for that Elastic IP and remove its email sending limitations. This process works well, and we haven’t had any problems sending mail from our mail server instance since completing it.

As with the general EC2 functions, EBS, ELB and Elastic IPs can be controlled via the AWS Management Console, but the CLI tools tend to be more powerful than the console. CLI commands for EBS and Elastic IPs are included within the EC2 command line tools, but ELB has its own set of CLI tools.

The actual migration of our infrastructure to EC2 was similar to other infrastructure migrations I’ve done in the past, but with the added bonus of EC2′s flexibility. This means that you can set up the infrastructure roughly how you think it will need to be, and then tweak it as you go along by starting and stopping instances, changing instance types, etc., instead of having to predict things as best you can then hope for the best. Since the migration, our entire infrastructure has been successfully running in the cloud for several months, and we continue to be very happy with Amazon Web Services. We closed our account with our colo provider some time ago, then sold all our physical servers.

For those of you who haven’t yet started using EC2, or who haven’t got very far with it yet, lots of useful details on the practicalities of hosting on EC2 can be found in this excellent series of blog posts written by my friend Paul Norman.

There’s a lot more you can do with EC2 which I haven’t covered, but hopefully this post gives an idea of how easy it is to set up a web infrastructure on there, and by reading the documentation and following the updates from Amazon, you can easily work out which additional details you need.