Event Counters

If you are unaware, almost every processor manufactured in recent history has some collection of event counters that are incremented when some processor event occurs. These events can range from clock cycles ticking by, instructions being retired, thermal thresholds being passed, or second level cache misses.

So far, I’ve only really used the CPU clock cycles, level 1 cache line replacement, and instructions retired event counters. Your needs might not match mine, so venture over to the Programmer Manual when you need something else!

Event Ratios

Related to the event counters are event ratios. These simple ratios can help you find specific performance issues in your program. For example, if your program does a lot of memory accesses, the processor may need to replaced cache lines frequently. But cache line replacements are naturally occurring in programs, how do we find excessive? Simple! We can just use the ratio of L1 cache replacements to the number of instructions retired. Then we’ll have an idea of how many times per instruction an L1 cache line is replaced.

Using oprofile

First, you’ll have to be running Linux, then you’ll want to install the “oprofile” package. Since this software installs kernel modules for monitoring, you’ll also need root/sudo access to allow the module to be loaded and unloaded for monitoring sessions. Here, I’ll be running as a user and using the sudo command when needed.

opcontrol

opcontrol is main program that lets you interact with the kernel. If you need a down-and-dirty list of the events available for monitoring, opcontrol --list-events will show you all the event counters at your disposal.

On my processor, the default event to monitor is CPU_CLK_UNHALTED which will tell me where the processor spent most of the time executing. If you want to monitor different events, you can specify what event(s) to monitor at the command line.

$ sudo opcontrol --event L1D_REPL:10000 --event INST_RETIRED:10000

The :10000 after each counter simply specifies what the trigger threshold is for raising processor exception. In other words, every 10,000 instructions retired the processor raises an exception that the oprofile daemon will catch and then increment the sample counter for that event. So, if you see that oprofile has 1 sample of the INST_RETIRED counter then the processor has seen 10,000 such events.

Now that we have the event counters configured, we can start the monitoring.

$ sudo opcontrol --start

Since the system is doing other activities, it is best if what you want to monitor can monopolize the system for the while. In my case I build a simple program that purposefully causes the L1 cache to have a lot of misses (l1thrash source code). I’ll also set the program to execute on one processor (CPU 1).

$ taskset 02 ./l1thrash

After it finishes executing, stop oprofile from running and save the profile session on the disk.

$ sudo opcontrol --stop
$ sudo opcontrol --save l1thrash

Now we have our profile saved to disk and we can view it with opreport.

opreport

Finally, we get to see how the program handled! Since we were smart and saved our profile to a session, we’ll have to specify that at the command line. You might want to pipe the output to less since it can be long at times. On my eight core system the output looks ugly.

$ opreport session:l1thrash
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted L1D_REPL events (Cache lines allocated in the L1 data cache) with a unit mask of 0x0f (No unit mask) count 10000
Samples on CPU 0
Samples on CPU 1
Samples on CPU 2
Samples on CPU 3
Samples on CPU 4
Samples on CPU 5
Samples on CPU 6
Samples on CPU 7
    cpu:0|            cpu:1|            cpu:2|            cpu:3|            cpu:4|            cpu:5|            cpu:6|            cpu:7|
  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|
------------------------------------------------------------------------------------------------------------------------------------------------
      541 95.7522      2969  0.9630       301 92.6154       484 69.9422       797 92.6744       707 88.2647       675 90.3614       707 89.8348 vmlinux
        7  1.2389        21  0.0068         6  1.8462         6  0.8671         9  1.0465         6  0.7491         6  0.8032         5  0.6353 oprofile
        6  1.0619         3 9.7e-04         6  1.8462         1  0.1445         3  0.3488         2  0.2497         1  0.1339         4  0.5083 nf_ses_watch
        5  0.8850        16  0.0052         6  1.8462         7  1.0116        25  2.9070        23  2.8714        30  4.0161        27  3.4307 libc-2.5.so
        3  0.5310         2 6.5e-04         1  0.3077         1  0.1445         2  0.2326         4  0.4994         0       0         1  0.1271 libpython2.4.so.1.0
        1  0.1770         0       0         0       0         0       0         0       0         0       0         0       0         0       0 e1000e
        1  0.1770         0       0         0       0         0       0         0       0         0       0         0       0         0       0 irqbalance
        1  0.1770         1 3.2e-04         1  0.3077         0       0         0       0         0       0         0       0         0       0 sshd
        0       0         2 6.5e-04         0       0         0       0         6  0.6977         8  0.9988         8  1.0710        18  2.2872 bash
        0       0         0       0         0       0         0       0         1  0.1163         0       0         0       0         1  0.1271 gawk
        0       0         0       0         0       0         3  0.4335         1  0.1163         2  0.2497         1  0.1339         0       0 bnx2
        0       0         0       0         3  0.9231         0       0         0       0         0       0         0       0         0       0 ehci_hcd
        0       0    305283 99.0179         0       0         0       0         1  0.1163         4  0.4994         1  0.1339         2  0.2541 l1thrash
        0       0        10  0.0032         0       0         0       0        14  1.6279        12  1.4981        19  2.5435        13  1.6518 ld-2.5.so
        0       0         3 9.7e-04         0       0         0       0         1  0.1163         1  0.1248         2  0.2677         2  0.2541 libcrypto.so.0.9.8b
        0       0         0       0         1  0.3077         0       0         0       0         0       0         0       0         0       0 libm-2.5.so
        0       0         0       0         0       0         0       0         0       0         0       0         1  0.1339         0       0 libpthread-2.5.so
        0       0         0       0         0       0         0       0         0       0         0       0         1  0.1339         0       0 syslogd
        0       0         0       0         0       0         0       0         0       0         1  0.1248         0       0         0       0 which
        0       0         0       0         0       0         1  0.1445         0       0         0       0         0       0         0       0 libcups.so.2
        0       0         0       0         0       0         0       0         0       0         0       0         2  0.2677         0       0 libusb-0.1.so.4.4.4
        0       0         0       0         0       0       189 27.3121         0       0        30  3.7453         0       0         7  0.8895 oprofiled
        0       0         1 3.2e-04         0       0         0       0         0       0         1  0.1248         0       0         0       0 cupsd

You may notice that the columns try to be sorted in descending order by the number of samples taken for a specific process. However, on CPU 1 (where we ran l1thrash) the sorted order isn’t close to correct. Luckily, we know that the bulk of our program only ran on CPU 1, so we can reissue the opreport command specifying that we only care about that processor.

$ opreport session:l1thrash cpu:1
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted INST_RETIRED.ANY_P events (number of instructions retired) with a unit mask of 0x00 (No unit mask) count 10000
Counted L1D_REPL events (Cache lines allocated in the L1 data cache) with a unit mask of 0x0f (No unit mask) count 10000
INST_RETIRED:1...|   L1D_REPL:10000|
  samples|      %|  samples|      %|
------------------------------------
  1834500 91.0882    305283 99.0179 l1thrash
   154499  7.6713      2969  0.9630 vmlinux
    21655  1.0752        21  0.0068 oprofile
     2176  0.1080        16  0.0052 libc-2.5.so
      442  0.0219        10  0.0032 ld-2.5.so
      435  0.0216         2 6.5e-04 bash
      108  0.0054         3 9.7e-04 libcrypto.so.0.9.8b
       47  0.0023         3 9.7e-04 nf_ses_watch
       43  0.0021         1 3.2e-04 sshd
       35  0.0017         2 6.5e-04 libpython2.4.so.1.0
       10 5.0e-04         0       0 libavahi-common.so.3.4.3
       10 5.0e-04         1 3.2e-04 cupsd
        9 4.5e-04         0       0 libcups.so.2
        7 3.5e-04         0       0 bnx2
        3 1.5e-04         0       0 libavahi-core.so.4.0.5
        1 5.0e-05         0       0 libpthread-2.5.so
        1 5.0e-05         0       0 timemodule.so

That looks better! Since we’ve narrowed down the output to one CPU, we now get to see both events that we monitored too. You can see that the majority of the time was spent in our l1thrash program, but how did it do?

We know that the number of samples is the number of times that the event counter on the processor hit 10,000 for both counters. So, we find that our l1thrash program caused level 1 cache replacements and retired instructions. Egads! Is that good or bad? Well, now we can throw in our ratio calculation for the L1 data cache miss:

That seems pretty bad to me! We can also see that the Linux kernel (vmlinux) had a ratio of 2,969:154,499 or about 1.9%, that is a fairly typical miss ratio.

A Second Example

This is a real example of a program I am actively trying to improve. The program is a kernel module (nf_ses_watch) designed to intercept packets at a decent rate, it is not performing well. Here I’ll use the default CPU_CLK_UNHALTED event monitor to see where the processor spends most of its time.

$ # I've already loaded the kernel module and started my packet generator
$ sudo opcontrol --event default
$ sudo opcontrol --start
$ # I'll wait about 30 seconds so there are enough samples to be meaningful
$ sudo opcontrol --stop
$ sudo opcontrol --save bombard

Now I have my saved session and can look at the profile. I’ve also taken the time to set the interrupt affinity of the Ethernet device to a specific processor (CPU 7), so now we can see if all the time was spent in my code of Linux code.

$ opreport session:bombard cpu:7
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
  samples|      %|
------------------
   737746 86.6169 nf_ses_watch
    88183 10.3533 vmlinux
    16810  1.9736 e1000e
     3680  0.4321 oprofiled
     2594  0.3046 oprofile
     1578  0.1853 libc-2.5.so
      900  0.1057 bash
       78  0.0092 ld-2.5.so
       52  0.0061 ophelp
       26  0.0031 libavahi-common.so.3.4.3
       22  0.0026 libavahi-core.so.4.0.5
       13  0.0015 gawk
        9  0.0011 libcrypto.so.0.9.8b
        9  0.0011 libpython2.4.so.1.0
        9  0.0011 sshd
        8 9.4e-04 bnx2
        4 4.7e-04 libpthread-2.5.so
        3 3.5e-04 grep
        2 2.3e-04 ipv6
        2 2.3e-04 auditd
        1 1.2e-04 cat
        1 1.2e-04 libdl-2.5.so
        1 1.2e-04 libm-2.5.so
        1 1.2e-04 libpcre.so.0.0.1
        1 1.2e-04 dirname
        1 1.2e-04 automount

Wow! Over 86% of the time we were executing code in the nf_ses_watch kernel module (my code)! Let’s see if we can dig a little deeper. First, oprofile has already done the work for us and tracks the specific symbol name within a piece of code that was active when the sample was taken with the --symbols option (this results in a very long list of kernel symbols). But, in the case of a kernel module, opreport doesn’t know where to find the symbol names so we have to tell it where the kernel module lives with --image-path.

$ opreport session:bombard cpu:7 --symbols --image-path ~/nf_ses_watch/kmod | head
warning: /bnx2 could not be found.
warning: /e1000e could not be found.
warning: /ipv6 could not be found.
warning: /oprofile could not be found.
warning: /sbin/auditd could not be read.
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
warning: could not check that the binary file /home/mjschultz/mon/module/kmod/nf_ses_watch.ko has not been modified since the profile was taken. Results may be inaccurate.
samples  %        image name               app name                 symbol name
733996   86.1767  nf_ses_watch.ko          nf_ses_watch             do_rip_entry
16810     1.9736  e1000e                   e1000e                   (no symbols)
10308     1.2102  vmlinux                  vmlinux                  rb_get_reader_page
9785      1.1488  vmlinux                  vmlinux                  read_hpet
8701      1.0216  vmlinux                  vmlinux                  ring_buffer_consume
3606      0.4234  vmlinux                  vmlinux                  netif_receive_skb
3530      0.4144  vmlinux                  vmlinux                  kfree

(I’ve piped the output through head to keep it reasonable.) We can see the real dirt here! By a huge margin, the do_rip_entry symbol in my nf_ses_watch module executes more than the Ethernet driver that is handling the raw packets. So that is where I’ll be looking when I try to resolve my bug.

Conclusions

If you are looking to optimize your program, oprofile is a great tool to use. The default event monitor (CPU clock cycles on most processors), can give you an idea of what part of your program is using most of the processor time. Once you know that, you can focus your efforts on reducing the number of cycles spent in that function. But don’t forget about all those other events too. If you have a memory intensive application, maybe you could reduce the memory contention and get an effective speedup with almost no refactoring!

(I’ve tried my best to be accurate with this information and I welcome any explicit corrections or clarifications.)

Now that I’m somewhat settled (and a little wiser), I decided to once more try my hand at a solid backup plan.  This was mainly motivated by a recent reinstall of my wife’s system (no lost data, just operating system upgrade).  Since I don’t have vast amounts of time on my hands, I didn’t want to forward port my old solution to get it to work on Linux and Mac OS X, so I looked for new solutions.  I recalled my advisor from Marquette mentioning rdiff-backup as what he put on his wife’s machine during her dissertation days.

As it turns out, rdiff-backup does most of what I wanted out of my backup solution and, in fact, does it a little better.  The main issue I had with my system was that it would periodically (monthly) take a snapshot of my home directory, after that it would periodically (weekly) build incremental diffs based off that snapshot.  What this boils down to is that, if a catastrophic failure happens I would roll back to the most recent snapshot, then progress forward in time to the most recent incremental file.  Not bad, but if you want better-than-weekly granularity it could be a lot of work.  Obviously, I had scripted this part, but still it is wasted time.  With rdiff-backup, it would be a single copy operation to restore to the most recent version.  If you wanted older versions you could roll back through the incremental diffs (again, it is automated).

The other feature that I needed was the ability to remove backups/incremental data older than some time frame (monthly).  Again, rdiff-backup gives me this ability at the command line.  Other bonuses include the fact that it is cross-platform (via macports or most Linux repositories), written in Python, and not maintained by me!

With the basic service in place, it was time to make it automated.  Again, linked off rdiff-backup’s page is an article on how to do unattended backups.  Besides the typical unattended SSH-keypair-without-a-passphrase and protecting-the-account steps, it introduced me to a new trick (which for some reason, despite having the knowledge on how to do it, never put together) using SSH config.

Host athena-backup
	Hostname athena.olympus
	User backups
	IdentityFile ~/.ssh/backups_rsa
	Compression yes
	Protocol 2

Now, if I try to ssh athena-backup, it’ll automatically use the correct identity file and user name which saves me from having to specify it on the command line (which you can’t typically do with wrapper functionality).  More importantly, it doesn’t break normal SSHing onto that host since we made it a special host (that’s the part I never put together, even though I knew it was possible).

The next issue I never took that time to think about before was my having moved from desktop to laptop (thereby making 1:00am backups worthless sense the laptop isn’t always on).  Because rdiff-backup does a roll-back model instead of my roll-forward model, I decided to do hourly backups to my home machine, thus I’ll likely catch at least one of these a day.  But I’m not always at home!  Getting around that is trivial, I’ll just ping the backup server before trying.  If it doesn’t respond, I don’t backup. This is done through:

ping -c1 -t1 $SERVER > /dev/null 2>&1

where $SERVER is just the name of the backup server.  It pings the host once with a timeout of 1 second, if it succeeds the backup continues; otherwise the script exits.

Of course, setting up the cronjob is as simple as:

0 */1 * * * $HOME/.crontab/rdiff-backup.sh

Hopefully this time around the backup solution is more robust than before.

Attachment: rdiff-backup.sh

Luckily, the solution is fairly straightforward.  Once you have SSH’d into a remote host (using ssh -X), you simply need to run firefox -no-remote. Of course you may want to tack on > /dev/null and an ampersand & to ignore the output and background the task. (Thanks to The Open Sourcer.)

With Firefox 2.x this behaviour was somewhat undocumented, but with Firefox 3.x, running firefox --help from the command line shows the -no-remote option.  It also seems that the default (i.e. -remote), is “documented” on Mozilla’s site for Remote Control of UNIX Mozilla.

If you wanted to make the -no-remote behaviour the default when SSH’d into remote machines, you could simply add a few lines to your bash profile to alias the firefox command.

# If we're forwarding X over SSH, make firefox execute on this machine
if [ -n "$SSH_CONNECTION" -a -n "$DISPLAY" ]; then
    alias firefox='firefox -no-remote'
fi

At least that is what I did.

Variables

The two main types of variables in a Makefile are “recursively expanded” (=; equal to) and “simply expanded” (:=; set equal to—thanks Algol).  Recursively expanded is by far the most common and, realistically, the most confusing.  This form of a variable will not perform the substitution until the last possible moment (lazy evaluation), so if you have the line SOURCES = ${CONFIG} demo.c, make will remember that you use ${CONFIG} until it must be known. So if the value of CONFIG changes, the newest version of CONFIG will be used when it is evaluated. Simple expansion occurs when the variable is declared (eager evaluation), if another variable (CFLAGS) is referenced in a simply expanded declaration it will be replaced with the current value of CFLAGS.

Targets

A target occurs on the left-hand side of a rule, typically this is the name of the file you want to build.  There are special cases of this, most commonly, “clean.”  Usually when someone wants to `make clean’ they want all the object files generated from a previous make to be removed.  However, if you were to create a file named `clean’ the rule would never execute because the file clean is up-to-date.  This can be remedied with by a PHONY target.

.PHONY: clean
clean:
    rm -f *.o

This creates a phony target that depends on clean, which tells make to ignore any files named clean.

More on Variables

The important “automatic” variables are talked about in the presentation ($@, $<, $^, $+, and $?).  Also useful is the % expansion variable.  For example, if there was the rule %.o: %.c in a Makefile, this will tell make that to make any file ending in .o will depend on the same file ending in .c (i.e. a rule foo.o: foo.c automatically exists). This will then execute the same commands (say gcc -m32 -Os -o foo.o foo.c) for all files ending in .o. This is a perfect example of why using automatic variables is a great idea.

Well, I think that is everything I have to say about Makefiles in a short amount of time. If you have any questions please feel free to post in the comments. I’ll try my best to answer them promptly!

(While most of the content here is off the top of my head, I did reference the GNU make page. They have everything you wanted to know about make.)

Attachment: Software Development using GNU/Linux (PDF)

Original Presentation

Although it may be less useful without the accompanying speaker, the original presentation is available.

Source code for the shell script

I have made the source code (backup.sh) available for download. In the top matter of the file describe how to add the script to your crontab.

Description of the script

For me, the best way to learn something is to take it line by line and that is what I’m going to do below. Naturally I will combine lines which are similar to save space. Since the target audience is someone who has never seen a shell script, some information may seem unimportant to you.

The concatenated source code that appears on this page may not agree with the source available for download. Odds are I decided the change was not worth updating this page, but made available for consumption as the script. If you notice something that is greatly different please contact me.

#!/bin/sh

Selection of a shell interpreter. This must be the first line in the file and be prefix with the ‘hash-bang’ (or ’sh-bang’ for short).

I use /bin/sh since it seems to be the most universal amongst systems. It should be noted that on many systems /bin/sh is that same as /bin/bash, I do not know if this means the script will not work in /bin/sh.

SNAPDIR=/var/snapshots/$USER
RMT_DIR="user@hostname:~/snapshots"
RMT_OPTIONS="-i $HOME/.ssh/id_dsa"

Setting some simple variables. Note that there are no spaces between the variable name, the equal sign, and the value; your script will not work with spaces between these three items.

Here I set the snapshot directory (where snapshots should be stored) to be /var/snapshots/$USER. $USER is a special variable that is the same as the user running the script.

Next are RMT_DIR and RMT_OPTIONS which are quoted. Quotes simply make sure spaces are included in the variable. Again, $HOME is a environmental variable that is always a user’s home directory.

RMT_CMD=$(which scp)
DATE=$(date +%Y%m%d)
TAR=$(which tar)
MKDIR=$(which mkdir)
CHMOD=$(which chmod)

This group of variables will run commands in a “sub-shell” before setting the variable name to the value. For example $(which scp) will execute which scp on the system and assign the value returned to RMT_CMD.

LAST_FULL=$(stat -f "%Dc %Sc" -t "%Y%m%d" ${SNAPDIR}/full-*.tar.gz \
            2> /dev/null| sort -n | tail -n1)
LAST_TS=$(echo ${LAST_FULL} | awk '{ print $1}')
LAST_DATE=$(echo ${LAST_FULL} | awk '{ print $2}')

In the final group of variables we use “pipes” which use the output of the first command as the input of the second command. For LAST_FULL we first stat files of the pattern “full-*.tar.gz” in the ${SNAPDIR} directory. (N.B. ${SNAPDIR} dereferences the SNAPDIR variable we set earlier. The curly braces are not strictly necessary, however I use them when referring to local variables.) For stat I am specifying that the output should be of the form “<timestamp> <YYYYMMDD>”, then sort the output using the number in the first column, and finally, take only the last file listed.

Once the last full snapshot time is known, we split it into two variables (LAST_TS and LAST_DATE), again using pipes and awk.

if [ ! -d ${SNAPDIR} ]; then
	${MKDIR} ${SNAPDIR}
	${CHMOD} go-rwx ${SNAPDIR}
fi

We’ll start by making sure the directory snapshots directory exists. If it doesn’t, make the directory and remove all permission from anyone not this user.

function incr {
	${TAR} czf ${SNAPDIR}/incr-${DATE}.tar.gz \
	       --exclude-from $HOME/.snap-exclude \
	       --listed-incremental=${SNAPDIR}/$USER-${LAST_DATE}.snar \
	       $HOME < /dev/null 2< /dev/null
 
	${CHMOD} go-rwx ${SNAPDIR}/incr-${DATE}.tar.gz
 
	if [ "${RMT_CMD}" != "" ]; then
		${RMT_CMD} ${RMT_OPTIONS} ${SNAPDIR}/incr-${DATE}.tar.gz \
		       ${SNAPDIR}/$USER-${LAST_DATE}.snar \
		       ${RMT_DIR}
	fi
}

For creating an incremental backup. Using tar, create a gzipped file at ${SNAPDIR}/incr-${DATE}.tar.gz. Since you may not want all of your home directory backed up, you can exclude files listed in the .snap-exclude file. Now the most important part, --listed-incremental, tells tar what the timestamps of the files were last time it executed. If the timestamp on a file is newer than in the snar (“snapshort archive”), it will be added to the tarball. The last argument to tar is simply the directory to backup. > and 2> redirect standard out and standard error to /dev/null, thus suppressing all output.

For the sake of security, we revoke all access from the file except for the current user.

The final step is to check of a RMT_CMD exists, if it does execute it. scp works well for this step, as would rsync.

function full {
	${TAR} czf ${SNAPDIR}/full-${DATE}.tar.gz \
	       --exclude-from $HOME/.snap-exclude \
	       --listed-incremental=${SNAPDIR}/$USER-${DATE}.snar \
	       $HOME < /dev/null 2< /dev/null
 
	${CHMOD} go-rwx ${SNAPDIR}/full-${DATE}.tar.gz
	${CHMOD} go-rwx ${SNAPDIR}/$USER-${DATE}.snar
 
	if [ "${RMT_CMD}" != "" ]; then
		${CMT_CMD} ${RMT_OPTIONS} ${SNAPDIR}/full-${DATE}.tar.gz \
		       ${SNAPDIR}/$USER-${DATE}.snar \
		       ${RMT_DIR}
	fi
}

Creating a full backup is not much different than an incremental backup. The only difference is the --listed-incremental file (tar will create a new snapshot archive), thus starting with a fresh backup and timestamps. The reason for this is explained in the “Recovery” section.

The rest of the function is mostly the same as an incremental backup.

function normal {
	# Make a full backup if no backup exists
	if [ ! -f ${SNAPDIR}/$USER-${LAST_DATE}.snar ]; then
		full;
	else
		ELAPSED=$(($(date +%s) - ${LAST_TS}))
		SNAP_FRAME=$((7 * 24 * 60 * 60 - 3600))
 
		# Check if it has been over a week since a full snapshot
		if [ ${ELAPSED} -gt ${SNAP_FRAME} ]; then
			# make a full snapshot
			full;
			# clean up files older than 4 weeks
			clean;
		else
			incr;
		fi
 
		unset ELAPSED SNAP_FRAME
	fi
}

Here we have the main “brain” of the program. It begins by making sure a backup exists, if one doesn’t the script makes a full backup. If at least one full backup exists, then we find out how long it has been since the last full backup and compare that to how frequently full backups should be made. SNAP_FRAME holds the frequency in which backups should be made (every 7 days * 24 hours / day * 60 minutes / hour * 60 seconds / minute (minus 1 hour for time delays)). If too much time has passed, create a full backup and clean out the old files. Otherwise just create an incremental backup.

function clean {
	true
}

The script isn’t perfect. I have yet to determine a good way to clean out old files (one that isn’t tied to either scp or rsync).

function usage {
	echo "usage: $0 [type]"
	echo "[type] can be one of the following:"
	echo "  normal - follow the daily incremental and weekly backup schedule"
	echo "  incr   - create a incremental backup of $HOME to ${SNAPDIR}"
	echo "  full   - create a full backup of $HOME to ${SNAPDIR}"
	echo "  clean  - cleanup backups older than one month"
	echo "  usage  - display this screen"
	echo "  --help - display this screen"
 
	exit 1
}

This simple function displays the usage information if requested. $0 is the script name as typed by the user.

case "$1" in
	'normal')
		normal;
		;;
	'incr')
		incr;
		;;
	'full')
		full;
		;;
	'clean')
		clean;
		;;
	'--help')
		usage;
		;;
	'usage')
		usage;
		;;
	*)
		normal;
		;;
esac

Finally, the driver of the program, a case statement which reads the first argument ($1) and executes the desired function. The default operation is to run in normal mode, but the user is able to force an incremental update, full update, or clean out old files.

Setting up a cronjob

cron is a simple utility that exists on almost all UNIX or UNIX-like systems. A daemon runs every minute to see if any user has a “cronjob” that needs to be executed, if a user does it will run it.

User level cronjobs are maintained by a program called crontab, to view your current crontab type: crontab -l, to edit your crontab use crontab -e.

I personally like to keep all my user level cronjobs in one place, $HOME/.cronjobs/. This folder conatins two files: crontab and backup.sh. crontab is a text file which hold what cronjobs I’d like to have run while backup.sh is the file described above.

My crontab files looks something like this:

# User level crontab
# min hr mday month wday command
00    13  *    *     *    /path/to/home/directory/.cronjobs/backup.sh

Which means I backup my files everyday at precisly 1:00pm by running the file located in /path/to/home/directory/.cronjobs/backup.sh. This can then be loaded into the system cronjobs using the following command.

crontab < crontab

Recovering the data

If you ever need to restore your backed up data all you need to do is find the most recent full backup (we’ll say full-20071001.tar.gz) and all the incremental backups since then (in our example incr-2007100[2-5].tar.gz). You’ll start by extracting the full backup to the correct folder via tar xzf full-20071001.tar.gz, followed by the incremental backups oldest to newest. Effectively you are restoring your entire home folder from n-days ago and applying the differences from each succeeding day. The commands should go as below (where $ is the shell prompt).

$ tar xzf full-20071001.tar.gz
$ tar xzf incr-20071002.tar.gz
$ tar xzf incr-20071003.tar.gz
$ tar xzf incr-20071004.tar.gz
$ tar xzf incr-20071005.tar.gz

After which you should have you home directory restored exactly as it appeared at 1:00pm on October 10, 2007.

Attachment: Automatic backups using cron and tar (PDF)
Attachment: backup.sh (Shell script)