I've found plenty of articles out there explaining how to use KVM with graphical GUI tools. On most of the CentOS servers I administer, however, I use Kickstart to create a customised and minimal GUI-free install to keep things as simple and efficient as possible. Here, therefore, are some guidelines for how to set up a virtualisation environment and virtual machines using KVM on CentOS 6 via the CLI.

I'm using LVM for the disk volumes because that's supposedly better for IO efficiency than using disk images stored on the host filesystem, so a bit of familiarity with LVM is ideally needed before getting started.

Initial host setup

Firstly it's necessary to make sure you have all the necessary software installed:

yum -y groupinstall Virtualization "Virtualization Client" "Virtualization Platform" "Virtualization Tools"
yum -y install libguestfs-tools

Next check that libvirtd is running:

service libvirtd status

If not, make sure that messagebus and avahi-daemon are running, then start libvirtd:

service messagebus start
service avahi-daemon start
service libvirtd start

Use chkconfig to ensure that all three of these services start automatically on system boot.

Next it's necessary to set up the network bridge so that the virtual machines can function on the network in the same way as physical servers. To do this, copy /etc/sysconfig/network-scripts/ifcfg-eth0 (or whichever is the file for the active network interface) to /etc/sysconfig/network-scripts/ifcfg-br0.

In /etc/sysconfig/network-scripts/ifcfg-eth0, comment out all the lines for BOOTPROTO, DNS1, GATEWAY, IPADDR and NETMASK, then add this line:

BRIDGE="br0"

Then edit /etc/sysconfig/network-scripts/ifcfg-br0, comment out the HWADDR line, change the DEVICE to "br0", and change the TYPE to "Bridge" (or add this line in if it's not here; note the capital B at the start of "Bridge").

Then restart the network:

service network restart

The bridge should now be up and running. You can check its status with:

ifconfig
brctl show

Creating the disk volumes for a new virtual machine

We need to create new LVM volumes for the root and swap partitions in the new virtual machine. I'm assuming LVM is already being used, that the volume group is called "sysvg", and that there is sufficient free space available in the sysvg group for the new volumes. If your volume group has a different name then just modify the instructions below accordingly. Change the volume sizes to suit your requirements:

lvcreate -L 20G -n vm-root sysvg
lvcreate -L 4096M -n vm-swap sysvg

Installing the operating system on the new virtual machine

Here I'm installing CentOS 6 on the guest machine using Kickstart, although I will also explain how to perform a normal non-automated installation. You'll need to modify the instructions accordingly to install different operating systems.

To make CentOS easily available for the installation, firstly make sure you have Apache installed and running. If necessary, install it with:

yum -y install httpd

Then start it with:

service httpd start

Then create the directory /var/www/html/CentOS and copy the contents of the CentOS DVDs into it.

If you're using Kickstart then you'll need these lines in your Kickstart config file to make sure that it can find the files from the CentOS DVDs. The IP address of the host in this example is 192.168.1.1, so change that as needed:

install
url --url http://192.168.1.1/CentOS

These lines are also required to make sure that Kickstart can find and use the LVM volumes created earlier:

zerombr
clearpart --all --initlabel
bootloader --location=mbr
part / --fstype ext4 --size 1 --grow --ondrive=vda
part swap --size 1 --grow --ondrive=vdb

Once the Kickstart file is ready, call it ks.cfg and copy it to /var/www/html.

This command installs CentOS on the guest using a Kickstart automated installation. The guest is called "vm", it has a dedicated physical CPU core (core number 2) and 1 GB of RAM allocated to it. Again, the IP address of the host is 192.168.1.1, so change that as needed:

virt-install --name=vm --cpuset=2 --ram=1024
  --network bridge=br0 --disk=/dev/mapper/sysvg-vm--root
  --disk=/dev/mapper/sysvg-vm--swap --vnc --vnclisten=0.0.0.0
  --noautoconsole --location /var/www/html/CentOS
  --extra-args "ks=http://192.168.1.1/ks.cfg"

The installation screen can be seen by connecting to the host via VNC. This isn't necessary for a Kickstart installation (unless something goes wrong). If you want to do a normal install rather than a Kickstart install then you will need to use VNC to get to the installation screen, and in that case you'll want to use the virt-install command above but just leave off everything from --extra-args onwards.

Also, you may want to install directly from a CDROM image, in which case replace the --location bit with --cdrom= and the path to the CD image, e.g. to install Ubuntu in your VM you might put --cdrom=/tmp/ubuntu-12.04.1-server-i386.iso.

(If virtual servers are already using VNC on the host then you will need to add the appropriate number to the VNC port number to connect to, e.g. the standard VNC port is 5900, and if there are already two virtual servers using VNC on the host then you will need to connect VNC to port 5902 for this install.)

General administration of virtual machines

Once you've got your virtual machine installed, you'll need to know the various commands for everyday administration of KVM virtual machines. In these examples, change the name of the VM from "vm" to whatever yours is called.

To show general info about virtual machines, including names and current state:

virsh list --all

To see a top-style monitor window for all VMs:

virt-top

To show info about a specific virtual machine:

virsh dominfo vm

To start a virtual machine:

virsh start vm

To pause a virtual machine:

virsh suspend vm

To resume a virtual machine:

virsh resume vm

To shut down a virtual machine (the 'acpid' service must be running on the guest for this to work):

virsh shutdown vm

To force a hard shutdown of a virtual machine:

virsh destroy vm

To remove a domain (don't do this unless you're sure you really don't want this virtual machine any more):

virsh undefine vm

Cloning virtual machines

To clone a guest VM, firstly it's necessary to create new disk volumes for the clone, then we use the virt-clone command to clone the existing VM:

lvcreate -L 20G -n newvm-root sysvg
lvcreate -L 4096M -n newvm-swap sysvg
virt-clone -o vm -n newvm -f /dev/mapper/sysvg-newvm--root -f /dev/mapper/sysvg-newvm--swap

Then dump the XML for the new VM:

virsh dumpxml newvm > /tmp/newvm.xml

Edit /tmp/newvm.xml. Look for the vcpu line and change the cpuset number to the CPU core you want to dedicate to this VM. Then make this change effective:

virsh define /tmp/newvm.xml

You'll also need to grab the MAC address from the XML. Keep this available as we'll need it in a minute:

grep "mac address" /tmp/newvm.xml | awk -F ' '{print $2}'

Start up the new VM and connect to it via VNC as per the instructions in the Installation section above. Edit /etc/sysconfig/network and change the hostname to whatever you want to use for this new machine. Then edit /etc/sysconfig/network-scripts/ifcfg-eth0 and change the HOSTNAME and IPADDR to the settings you want for this new machine. Change the HWADDR to the MAC address you obtained a moment ago, making sure that the letters are capitalised.

Then reboot and the new VM should be ready.

Backing up and migrating virtual machines

In order to take backups and to be able to move disk volumes from virtual machines to other hosts, we basically need to create disk image files from the LVM volumes. We'll first snapshot the LVM volume and take the disk image from the snapshot, as this significantly reduces the amount of time that the VM needs to remain paused (i.e. effectively offline) for. We remove the snapshot at the end of the process so that the VM's IO is not negatively affected.

This disk image, once created, can then be stored in a separate location as a backup, and/or transferred to another host server in order to copy or move the VM there.

So, make sure that the VM is paused or shut down, then create a LVM snapshot, then resume the VM, then create the image from the snapshot, then remove the snapshot:

virsh suspend vm
lvcreate -L 100M -n vm-root-snapshot -s /dev/sysvg/vm-root
virsh resume vm
dd if=/dev/mapper/sysvg-vm--root--snapshot of=/tmp/vm-root.img bs=1M
lvremove /dev/mapper/sysvg-vm--root--snapshot

You can then do what you like with /tmp/vm-root.img - store it as a backup, move it to another server, and so forth.

In order to restore from it or create a VM from it on a new server, firstly use lvcreate to create the LVM volume for restore if it isn't already there, then copy the disk image to the LVM volume:

dd if=/tmp/vm-root.img of=/dev/mapper/sysvg-vm--root bs=1M

You may also need to perform this procedure for the swap partition depending on what you are trying to achieve.

You'll also want to back up the current domain configuration for the virtual machine:

virsh dumpxml vm > /tmp/vm.xml

Then just store the XML file alongside the disk image(s) you've taken.

If you're moving the virtual machine to a new server then once you've got the root and swap LVM volumes in place, you'll need to create the domain for the virtual machine on the new server. Firstly edit the XML file and change the locations of disk volumes to the layout on the new server if it's different to the old server, then define the new domain:

virsh define /tmp/vm.xml

You should then be able to start up the "vm" virtual machine on the new server.

Resizing partitions on a guest

Let's say we want to expand the root partition on our VM from 20G to 25G. Firstly make sure the VM is shut down, then use virt-filesystems to get the information we need for the resize procedure:

virsh shutdown vm
virt-filesystems -lh -a /dev/mapper/sysvg-vm--root

This will probably tell you that the available filesystem on that volume is /dev/sda1, which is how these tools see the virtual machine's /dev/vda1 partition. We'll proceed on the basis that this is the case, but if the filesystem device name is different then alter the command below accordingly.

Next we create a new volume, then we perform the virt-resize command from the old volume to the new volume, then we set the new volume as the active root partition for our domain:

lvcreate -L 25G -n vm-rootnew sysvg
virt-resize --expand /dev/sda1 /dev/mapper/sysvg-vm--root /dev/mapper/sysvg-vm--rootnew
lvrename /dev/sysvg/vm-root /dev/sysvg/vm-rootold
lvrename /dev/sysvg/vm-rootnew /dev/sysvg/vm-root
virsh start vm

Then, when you're sure the guest is running OK with the new resized partition, remove the old root partition volume:

lvremove /dev/mapper/sysvg-vm--rootold