Useful Unix commands

For a long time I’ve maintained a memory aid in the form of a list of useful commands which can be used on the command line for Linux, OS X, BSD, Solaris, etc., so I thought I’d list them in a blog post in case they come in useful for others. Most of these will run on any Unix-type operating system, though I’ve usually indicated where a command is OS-specific.

  • For Debian/Ubuntu distributions, many of these commands are available via the APT package manager.
  • For Red Hat/CentOS/Fedora, many of these commands are available via the yum package manager, though particularly in the case of CentOS I’d recommend adding the EPEL repository to increase the availablility of useful tools which are otherwise missing.
  • For OS X, install the excellent Homebrew package manager, then many of these commands will be available for install.

This is a fairly arbitrary list which I add to when I forget to use something old, or when I come across something new which is useful. Some of these commands will probably be familiar to you, and some probably won’t. I’ve added links where applicable. Please feel free to throw amendments or additional suggestions my way.

The list of commands

  • ab – website benchmarking; usually comes with Apache HTTP server
  • afconvert – powerful audio file converter; OS X only
  • cpulimit – provides a simple way of limiting CPU usage for specific processes
  • curl – powerful URL transfer tool for testing web pages and other services
  • dc – CLI-based calculator
  • ditaa – converts ASCII art to PNG
  • dmidecode – reports system hardware as described in the BIOS
  • exiftool – for manipulating Exif data on image files
  • fio – IO benchmarking tool
  • goaccess – simple and powerful web log analyser and interactive viewer
  • html2text.py – converts HTML to Markdown; this is not the html2text which comes with e.g. Homebrew
  • htop – like top but nicer and more informative
  • http-server – start a web server in the current directory on port 8001, using Node.js; package needs to have been installed using: npm install http-server -g
  • httping – ping a host using HTTP instead of ICMP
  • iftop – like top but for network traffic
  • ike-scan – find and probe IPSec VPNs
  • iotop – like top but for disk IO
  • jp2a – converts JPEGs to ASCII art
  • lshw – simple and powerful way of getting hardware info; Linux only
  • lsof – for finding which processes are using which files and network ports, amongst other things
  • mitmproxy/mitmdump – nice HTTP sniffer proxy; usually installed via pip
  • mtr – handy graphical combination of ping and traceroute
  • mountpoint – check whether a directory is a mount point
  • multitail – tail multiple log files in separate panes in the same window
  • ncat – like nc/netcat but newer and with extra options; comes with nmap
  • nethogs – quick real-time display of how much bandwidth individual processes are using; Linux only
  • ngrep – for intelligently sniffing HTTP and other protocols
  • nl – add line numbers to input from file or stdin
  • nmap – comprehensive port/vulnerability scanner
  • nping – advanced ping tool for TCP/HTTP pinging; comes with nmap
  • opensnoop – watch file accesses in real time; OS X only
  • parallel – like xargs but better
  • paste – merge multi-line output into single line with optional delimiters
  • pen – simple but effective command line load balancer
  • pgrep/pkill – easy grepping for/killing of processes using various criteria
  • photorec – recover lost documents, videos, photos, etc. from storage media
  • pidstat – flexible tool for obtaining statistics on processes, very useful for understanding resource usage for particular processes
  • psk-crack – for cracking VPN preshared keys obtained with ike-scan; comes with ike-scan
  • printf – for reformatting; very useful for things like zero padding numbers in bash
  • pstree – shows a tree of running processes
  • pv – provides a progress bar for piped processes
  • python -m SimpleHTTPServer – start a web server in the current directory on port 8000, using Python
  • qlmanage -p – Quick Look from the command line; OS X only
  • s3cmd – CLI tool for Amazon S3 administration
  • scutil – for changing system settings including various forms of hostname; OS X only
  • seq – generates a sequence of numbers
  • siege – website benchmarking; more options than ab
  • sslscan – see which ciphers and SSL/TLS versions are supported on secure sites
  • stress – to artificially add load to a system for testing purposes
  • subnetcalc – IPv4/IPv6 subnet calculator
  • tcptraceroute – like traceroute but TCP
  • tee – for directing output to a file whilst watching it at the same time
  • time – gives info on how long a command took to run
  • timeout – run a command with a time limit, i.e. kill it if it’s still running after a certain time
  • tmutil – get more control over Time Machine from the command line; OS X only
  • tree – file listing in tree format
  • trickle – simple but effective bandwidth throttling
  • trimforce – turns on TRIM support for third-party SSDs; OS X only
  • watch – prepend to a command line to see continously updating output
  • wget – nice client for web downloads and archiving websites
  • xmlstarlet – powerful tool for manipulating and reformatting XML

How to use Flickr favourites as your screensaver in OS X Yosemite, Mavericks and Mountain Lion

Mountain Lion was an improvement on Lion, which I had mixed feelings about when it was released. Unfortunately, however, Apple apparently decided that RSS is a dead technology (although it seems to be creeping back into Safari), and consequently the handy RSS screensavers were removed, which means there was no simple way of creating a screensaver out of one’s Flickr favourites in Mountain Lion, and this remains the case in Mavericks and Yosemite.

Having come up with an effective solution for how to get Flickr favourites as a screensaver in Mountain Lion which also works in Mavericks and Yosemite, I thought I’d share the method for the benefit of those who are not so used to fiddling with the deeper technological aspects of their Mac. I’ve gone into quite a lot of detail for those who are less technically-minded, but those of a more technical bent can just skip ahead accordingly.

Yahoo! keeps changing the Flickr website and API which keeps breaking some of my instructions, so I’m constantly modifying this post so that it stays up to date for getting your favourites from the new Flickr website to use as a screensaver in Yosemite, Mavericks and Mountain Lion. I last updated these instructions on 28th October 2014 and they tested out fine on that date.

Set up the RSS feed

Thanks to Rudie for emailing me to point out that Flickr has apparently started providing large versions of images in its RSS feeds, so this part is now much simpler than it was before.

Firstly you need to get your Flickr ID. I’ve gone back to using idGettr for this after it stopped working for a while (presumably also due to constant Flickr API changes) but then became operational again. So, follow the simple instructions at idGettr, and what you need to take note of is the Flickr ID, which looks something like 39653492@N07.

Then simply add that to the end of the Flickr photo RSS feed address so you end up with something like this, but with your own Flickr ID at the end instead of mine:

https://api.flickr.com/services/feeds/photos_faves.gne?id=39653492@N07

Paste this link into a temporary place so you can come back to it shortly – a temporary document in TextEdit is as good a place as any.

Create the folders you need on your Mac

From here on it is necessary to venture into the dark underbelly of OS X via the use of the Terminal. So, launch the Terminal app using your preferred launch method. If you’re not sure how to do this, type ⌘-Space to bring up Spotlight, then type “terminal”, then choose “Terminal” in “Applications”, then hit Enter to launch it. Then we make sure we have the folders we will need, so type (or copy and paste) the following line into the Terminal:

mkdir -p ~/Pictures/Flickr_faves ~/Library/Scripts ~/Library/LaunchAgents

Create the script to download the images

The first thing we need to do next is determine what your short username is. On OS X, every user has a long username and a short username. For example, my long username might be “Matt Brock” and my short username might be “mattbrock”. If you already know for sure what your short username is then that’s great. If not, then just type the following into the Terminal and it will tell you:

echo $USER

Make a note of this somewhere if you’re not sure you can remember it. Next we create the script which will get the latest favourites and clear out old ones, so type:

nano ~/Library/Scripts/get_Flickr_faves.sh

This will bring you into a text editor. Paste in the following text:

#!/bin/bash

if ! cd ~/Pictures/Flickr_faves ; then
  logger "get_Flickr_faves: failed to cd to ~/Pictures/Flickr_faves; exiting"
  exit 1
fi

if ! curl -s "http://api.flickr.com/" > /dev/null 2>&1 ; then
  logger "get_Flickr_faves: couldn't connect to Yahoo API; exiting"
  exit 1
fi

curl -s "URL" | grep 'rel="enclosure"' | awk -F '"' '{print $6}' | xargs -L1 curl -s -O

find . -mtime +1 -exec rm -f {} +

NO_IMAGES=$(ls | wc -l | sed "s/ //g")

logger "get_Flickr_faves: completed; $NO_IMAGES images"

Then use the arrow keys on your keyboard to move around. You need to replace URL with the long RSS feed address which you pasted into a temporary location earlier – you can use regular copy and paste, plus the arrow keys and the delete key to do all this. Make sure you don’t change anything else, and make sure you keep the quotation marks around the RSS feed address. Once that’s done, type Ctrl-O, then Enter to confirm, to save the changes. Then type Ctrl-X to exit the text editor. Then type the following to give your new script permission to run:

chmod 755 ~/Library/Scripts/get_Flickr_faves.sh

You can now run the script once to download your Flickr favourites:

~/Library/Scripts/get_Flickr_faves.sh

Hopefully that won’t produce any errors, and if you go into Finder to look in the “Flickr_faves” folder in your Pictures folder, you should see local copies of your Flickr favourites in there.

Tell your Mac to update the images once per day

Next we need to tell your Mac to run the script we just created once a day in order to update the images from your favourites on Flickr. So, run the following command:

nano ~/Library/LaunchAgents/uk.co.mattbrock.Flickr_faves.plist

This will again bring you into a text editor. This time, paste in the following:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>get_Flickr_faves</string>
  <key>Program</key>
  <string>/Users/USERNAME/Library/Scripts/get_Flickr_faves.sh</string>
  <key>StartCalendarInterval</key>
  <dict>
    <key>Minute</key>
    <integer>15</integer>
    <key>Hour</key>
    <integer>15</integer>
  </dict>
  <key>Debug</key>
  <true/>
  <key>AbandonProcessGroup</key>
  <true/>
</dict>
</plist>

N.B. This assumes that your home directory is in /Users. If you’ve moved your home directory then you’ll need to alter the Program string to point to the alternative location instead of /Users/USERNAME. If you don’t know what this means then ignore it because it almost certainly doesn’t apply to you.

Replace USERNAME with your short username, then type Ctrl-O, Enter, then Ctrl-X to save and exit the editor. This is a set of instructions to tell your Mac to run the script we created earlier once a day in order to make sure your Flickr favourites are kept up to date locally. In order to get your Mac to read and act upon these instructions, we need to run the following:

launchctl load ~/Library/LaunchAgents/uk.co.mattbrock.Flickr_faves.plist

Hopefully that won’t produce any errors. We’re almost there now.

Configure the screensaver on your Mac

So all we need to do now is point the slideshow screensaver at the folder we created earlier to store the images from your Flickr favourites. Go into Desktop & Screen Saver in your System Preferences, click the Screen Saver tab, then choose one of the Slideshows (I prefer “Classic”), then under “Source” choose “Choose Folder…”, then navigate to the Flickr_faves folder in your Pictures folder. It’s up to you whether you choose “Shuffle slide order” – personally I prefer to have that on, as I like a bit of randomness. Then choose a time under “Start after” to determine how soon you’d like the screensaver to start up. Alternatively/additionally, you can click Hot Corners and choose a hot corner to trigger your new screensaver on demand. And that should be it! If all has gone well, your screensaver will now show you all your latest Flickr favourites. Enjoy.

Troubleshooting

You should be able to see the results of this automated script by launching Terminal and then typing:

grep get_Flickr_faves /var/log/system.log

This should show you the times when the task has run and what happened when it tried. When it’s been successful you will see something like the following:

Sep  4 15:17:56 mymac me[27308]: get_Flickr_faves: completed; 37 images

P.S. You can now get the Bash script and launchd XML configuration from GitHub if you so desire.

Creating a two-node CentOS 6 cluster with floating IP using Pacemaker and Corosync

These are my old instructions for creating a Linux cluster with floating IP on versions of CentOS prior to 6.4.

If you’re using CentOS 6.4 or higher, you need my updated post for creating a cluster with CMAN and Pacemaker.

Installation and configuration

Install the required packages and prepare the configuration file:

yum install pacemaker
cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf

Change bindnetaddr to your network address (e.g. 192.168.1.0) in
/etc/corosync/corosync.conf. Create /etc/corosync/service.d/pcmk and put
the Pacemaker startup config into it:

service {
        name: pacemaker
        ver: 1 
}

Start corosync and pacemaker:

service corosync start
service pacemaker start

Make sure they start on boot:

chkconfig corosync on
chkconfig pacemaker on

Repeat all of the above on the second node. Then disable STONITH on the primary node:

crm configure property stonith-enabled=false

Add a floating IP on the primary node (change IP address and netmask as
needed):

crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 params ip=192.168.1.110 cidr_netmask=24 op monitor interval=30s

Disable quorum on the primary node (necessary for running only two nodes):

crm configure property no-quorum-policy=ignore

Prevent resources from moving back after a node recovers:

crm configure rsc_defaults resource-stickiness=100

Use the commands in the Administration section below to ensure that the
floating IP is on the correct machine – if this is not the case then
temporarily shut down Pacemaker on whichever machine has the floating IP so
that it moves across to the desired node:

service pacemaker stop

Administration

Display the current configuration:

crm configure show

Status monitor:

crm status

Show which node has the cluster IP:

crm resource status ClusterIP

Reset the config to default:

cibadmin -E --force

Verify that there are no problems with the current config:

crm_verify -L

Creating a two-node CentOS 6 cluster with floating IP using CMAN and Pacemaker

Originally I was using Heartbeat to create two-node Linux clusters with floating IPs, but when Heartbeat stopped being developed I needed to figure out how to use Corosync and Pacemaker for this instead. Somewhat annoyingly, Linux HA stuff has changed yet again in CentOS 6.4, so now it’s necessary to use CMAN and Pacemaker instead.

This is quite a lot more in-depth than the simple configuration that was originally required for Heartbeat. Anyway, based on my recent experiences, here’s a very quick guide for if you find yourself in a similar situation. This works for me on CentOS 6.4 and higher, but it won’t work on earlier versions of CentOS.

If you’re looking for the old instructions for creating a cluster with Pacemaker and Corosync, they’re here.

Installation and initial configuration

Install the required packages on both machines:

yum install pacemaker
cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf

Set up and configure the cluster on the primary machine, changing newcluster , server1.example.com and server2.example.com as needed:

ccs -f /etc/cluster/cluster.conf --createcluster newcluster
ccs -f /etc/cluster/cluster.conf --addnode server1.example.com
ccs -f /etc/cluster/cluster.conf --addnode server2.example.com
ccs -f /etc/cluster/cluster.conf --addfencedev pcmk agent=fence_pcmk
ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect server1.example.com
ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect server2.example.com
ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk server1.example.com pcmk-redirect port=server1.example.com
ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk server2.example.com pcmk-redirect port=server2.example.com

Copy /etc/cluster/cluster.conf from the primary machine to the secondary machine in the cluster.

It’s necessary to turn off quorum checking, so do this on both machines:

echo "CMAN_QUORUM_TIMEOUT=0" >> /etc/sysconfig/cman

Start the services

Start up the services on both machines:

service cman start
service pacemaker start

Make sure both services start on reboot:

chkconfig cman on
chkconfig pacemaker on

Configure and create floating IP

Configure the cluster on the primary machine:

pcs property set stonith-enabled=false
pcs property set no-quorum-policy=ignore

Create the floating IP on the primary machine, changing the IP address and server name as needed:

pcs resource create livefrontendIP0 ocf:heartbeat:IPaddr2 ip=192.168.100.100 cidr_netmask=32 op monitor interval=30s
pcs constraint location livefrontendIP0 prefers server1.example.com=INFINITY

Cluster administration

To monitor the status of the cluster:

pcs status

To show the full cluster configuration:

pcs config