Now that I’m somewhat settled (and a little wiser), I decided to once more try my hand at a solid backup plan.  This was mainly motivated by a recent reinstall of my wife’s system (no lost data, just operating system upgrade).  Since I don’t have vast amounts of time on my hands, I didn’t want to forward port my old solution to get it to work on Linux and Mac OS X, so I looked for new solutions.  I recalled my advisor from Marquette mentioning rdiff-backup as what he put on his wife’s machine during her dissertation days.

As it turns out, rdiff-backup does most of what I wanted out of my backup solution and, in fact, does it a little better.  The main issue I had with my system was that it would periodically (monthly) take a snapshot of my home directory, after that it would periodically (weekly) build incremental diffs based off that snapshot.  What this boils down to is that, if a catastrophic failure happens I would roll back to the most recent snapshot, then progress forward in time to the most recent incremental file.  Not bad, but if you want better-than-weekly granularity it could be a lot of work.  Obviously, I had scripted this part, but still it is wasted time.  With rdiff-backup, it would be a single copy operation to restore to the most recent version.  If you wanted older versions you could roll back through the incremental diffs (again, it is automated).

The other feature that I needed was the ability to remove backups/incremental data older than some time frame (monthly).  Again, rdiff-backup gives me this ability at the command line.  Other bonuses include the fact that it is cross-platform (via macports or most Linux repositories), written in Python, and not maintained by me!

With the basic service in place, it was time to make it automated.  Again, linked off rdiff-backup’s page is an article on how to do unattended backups.  Besides the typical unattended SSH-keypair-without-a-passphrase and protecting-the-account steps, it introduced me to a new trick (which for some reason, despite having the knowledge on how to do it, never put together) using SSH config.

Host athena-backup
	Hostname athena.olympus
	User backups
	IdentityFile ~/.ssh/backups_rsa
	Compression yes
	Protocol 2

Now, if I try to ssh athena-backup, it’ll automatically use the correct identity file and user name which saves me from having to specify it on the command line (which you can’t typically do with wrapper functionality).  More importantly, it doesn’t break normal SSHing onto that host since we made it a special host (that’s the part I never put together, even though I knew it was possible).

The next issue I never took that time to think about before was my having moved from desktop to laptop (thereby making 1:00am backups worthless sense the laptop isn’t always on).  Because rdiff-backup does a roll-back model instead of my roll-forward model, I decided to do hourly backups to my home machine, thus I’ll likely catch at least one of these a day.  But I’m not always at home!  Getting around that is trivial, I’ll just ping the backup server before trying.  If it doesn’t respond, I don’t backup. This is done through:

ping -c1 -t1 $SERVER > /dev/null 2>&1

where $SERVER is just the name of the backup server.  It pings the host once with a timeout of 1 second, if it succeeds the backup continues; otherwise the script exits.

Of course, setting up the cronjob is as simple as:

0 */1 * * * $HOME/.crontab/rdiff-backup.sh

Hopefully this time around the backup solution is more robust than before.

Attachment: rdiff-backup.sh

http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul

It looks like someone likes Ghostbusters at Mozilla.  I just found it amusing.

Now—interesting as the talk was—he suggested everyone read his only non-technical paper titled, “Technology and Courage” [PDF from Sun].  It took me until yesterday to read it, but it was certainly an interesting article.  I recommend everyone read it.

I begin searching online for something that will watch RSS feeds on my behalf and send me an email when it updates.  The first thing I come across is Feed My Inbox, they seem to offer the exact service I want.  Upon closer inspection, they promise to only send one email every 24 hours.  That won’t cut it—I want my updates and I want them now!  After a bit more searching I find rss2email, a simple Python program that keeps track of multiple RSS feeds and converts entries into emails when it executes.  I go through the initial configuration and set up a cronjob to check for new entries every 4 minutes.  Good enough for now.

However, this brings up an annoyance with RSS feeds:

RSS feeds do not provide real-time updates

Once you get down to it, all an RSS feed does is provide some subset of content on a page.  It is still up to the client to ask the server when new content exists.  This fact has bugged me very slightly in the past since I know xkcd updates every Monday, Wednesday, and Friday at 11pm central time, but my RSS subscription in Firefox won’t provide me with the link until (at its discretion) polls the server for new content.  Now, it bothers me slightly more since I know I have to wait at least 4 minutes for my cronjob to run, plus any time it might take Google to publish the updates in the feed (but I’ll ignore that part).

Is there any way to improve this and give end-users real-time updates from content-providers?  It seems like this would be great for users of Twitter and Facebook, since the end-users want to know what is happening now.

One answer seems to be in push notifications, that seem to have been popularized by Blackberry and iPhone applications.  These allow a central server to send a tiny message to the phone that nudges the device that there is data to be had.  Of course, this works very well on phone systems that can easily associate an content generator with a telephone number to contact.  However, it is a bit tougher with IP-only devices that migrate from network to network.  Although, it seems Apple’s Push Notification Service (APNS) should be able to do this.  Either way, this technology seems to be heading in the right direction.

APNS works by maintaining a connection between the client and server, that way when an event happens server side it just sends it on to all the connected clients.  Unfortunately, I’m not convinced at how well these push notifications will scale.  A system implementing IP push notifications seems like it could easily have on the order of 1000s of simultaneous, persistent connections.  According to this “Comparison of Push and Pull Techniques for AJAX,” (tech report, PDF) push-style system do bog down servers a bit.  (Admittedly, the methodology for that paper might not be the best, but I’m guessing the conclusions are valid—I would like to see more/better studies of this.)

This bring me to what I want to see implemented or for someone to point me to the implementation of a distributed content syndication protocol (DCSP).  The high-level view that I think would work (I haven’t thought long or carefully about it), would be similar to other distributed networks.  The content provider would maintain a complete list of current computers subscribed to the feed and the feed itself.  The client would run software that asks the server who to connect to and select a few peers and create a long running connection.  When new content arrives, the server pushes the content to its peers, who push to their peers, and so forth.  This removes the burden of pushing content to all subscribers from the server, giving it scalability (in my mind).  It would then be up to the client to connect and maintain connections with a collection of peers to get the real-time updates.  I’m sure there would have to be some control messages to prevent flooding.  But, it seems like it would give real-time updates to users.

I suppose this would mean I would have to run another program on my system, but it could either be a front-end client that handles my content feeds or a daemon running in the background and set up to deliver an email to a local mailbox (or even a remote mailbox) when fresh content arrives.

Ah well, who knows if it would work.  Hell, maybe I just missed a fact about RSS that doesn’t make it suck as much as I think it does.  Thus ends my stream of though.

Luckily, the solution is fairly straightforward.  Once you have SSH’d into a remote host (using ssh -X), you simply need to run firefox -no-remote. Of course you may want to tack on > /dev/null and an ampersand & to ignore the output and background the task. (Thanks to The Open Sourcer.)

With Firefox 2.x this behaviour was somewhat undocumented, but with Firefox 3.x, running firefox --help from the command line shows the -no-remote option.  It also seems that the default (i.e. -remote), is “documented” on Mozilla’s site for Remote Control of UNIX Mozilla.

If you wanted to make the -no-remote behaviour the default when SSH’d into remote machines, you could simply add a few lines to your bash profile to alias the firefox command.

# If we're forwarding X over SSH, make firefox execute on this machine
if [ -n "$SSH_CONNECTION" -a -n "$DISPLAY" ]; then
    alias firefox='firefox -no-remote'
fi

At least that is what I did.

With mutt I choose to download all my email via POP3 to a local machine where I can read it when I get around to it (rigorous, isn’t it). After I read a message and deem it complete I move it to a folder named after the sender (or possibly a group) where I can grep the files and read them at a later date.

However, after a while these files pile up and I need to periodically compress and archive them. This, of course, gets annoying and frequently forgotten. To solve this I needed a script that could parse messages in a number of mail formats, find a date, and determine if it is beyond some threshold at which point it should be archived. These requirements brought me to archivemail. Archivemail supports several input formats (IMAP, mh, mbox, Maildir), archives the messages, and outputs a single mbox formatted file (that can be compresses). While I’m not a huge fan of the mbox format I can easily deal with it for archived mail.

Archivemail has several perks that fit my requirements quite well. First, it was easy to get (packages availables on OS X, Fedora, Debian, and Ubuntu), this probably stems from the fact that it is written in python and can easily run on almost any system. Next, it provides several useful command line options, I personally have a cronjob that archives four message folders every 30 days (logwatches and mail lists) and archives other messages after 180 days. This is simply done with the --days command line switch. I also specify a directory to dump all the archived messages into so they don’t clutter up my mail directory. Depending on how you handle you mail there are also options to not archive unread messages or only archive messages older than some fixed date.

For those interested, here is my script that I run as a weekly cronjob to archive and compress my mail messages:

ARCMAIL="/usr/bin/archivemail --quiet --output-dir=$HOME/mail/archive/"

$ARCMAIL --days  30 $HOME/mail/logwatch \
                    $HOME/mail/netflix  \
                    $HOME/mail/amazon   \
                    $HOME/mail/dreamhost

$ARCMAIL --days 180 $HOME/mail/*

Fairly straightforward, eh?

To search through an archive you can just change into the archive/ directory and execute a gunzip -c <filename> | grep <word>. Alternatively, you can use mutt’s built in search and run gunzip <filename>.gz ; mutt -f <filename>.

So, I’ve got Bazaar down pretty well—it’s design has been engineered for a smooth transition from Subversion—but Git is a different beast.

Git breaks a lot of the conceptual assumptions made in svn-land that you didn’t even know you were using to understand your daily VCS use; the basic “checkout, update, commit” operations don’t cleanly map to the git paradigm. This can create many WTF moments spent staring at lines of help like:

git-rebase - Forward-port local commits to the updated upstream head

The payoff for diving into these treacherous waters is, IMHO, worth it. When your first ‘git commit’ or ‘git checkout’ comes back within a few milliseconds and you’re left sitting there still trying to believe it’s already done… this is when you realize “Wow, this tool might actually change the way I work.”

  OPS = {
    "="  =>  lambda { |v, r| v == r },
    "!=" =>  lambda { |v, r| v != r },
    ">"  =>  lambda { |v, r| v > r },
    "<"  =>  lambda { |v, r| v < r },
    ">=" =>  lambda { |v, r| v >= r },
    "<=" =>  lambda { |v, r| v <= r },
    "~>" =>  lambda { |v, r| v = v.release; v >= r && v < r.bump }
  }

Found deep in the rubygems source.

First of all, the reason I was using YAML was as a quick, simple, human-readable backend for a tiny little side-project. I chose YAML over SQLite or CSV because neither have the human-readable thing going for them, plus I’m working in Ruby and it seemed to have nice built-in support for YAML.

You can take any object you like (with obvious exceptions like streams and sockets) and just YAML.dump it, creating a happy string waiting to be written to the nearest file. So given this class:

class Foo
  def initialize
    @bar = 'one'
    @baz = 2
    @qux = [3, 'four', 5]
  end
end

You can do something like this:

>> f = Foo.new
=> #
>> YAML.dump(f)
=> "--- !ruby/object:Foo \nbar: one\nbaz: 2\nqux: \n- 3\n- four\n- 5\n"

And that YAML string at the end there looks like this:

--- !ruby/object:Foo
bar: one
baz: 2
qux:
- 3
- four
- 5

So you can stuff that wherever you’d like (file.yml let’s say) and then at some later date all you need to do is:

>> new_f = YAML.load_file('file.yml')
=> #
>> f == new_f
=> true

Fun times, no? And most importantly fun times that don’t require one learns YAML in great detail. Well, let me show you exactly where the fun times ended for me. For an example, let’s make a simple hash with our f.

>> h = { f => 2 }
=> {#=>2}
>> YAML.dump(h)
=> "--- \n!ruby/object:Foo ? \n  bar: one\n  baz: 2\n  qux: \n  - 3\n  - four\n  - 5\n: 2\n\n"

Seems fine, no?

---
!ruby/object:Foo ?
  bar: one
  baz: 2
  qux:
  - 3
  - four
  - 5
: 2

So we go about our business, right?

>> YAML.load(YAML.dump(h))
ArgumentError: syntax error on line 2, col -1

Boo. So now I have to go and do what I was trying to avoid doing in the first place: learn about YAML. Well after wading through the schema documents for YAML, which are long, full of BNF, and no fun. At the end of the day it turns out that YAML.dump produces invalid YAML whenever an object is used as a key in a Hash. Without going into too much detail (partially because I don’t fully understand it), the question mark needs to come before the ruby object tag for it to be valid.

Here’s a workaround that just does some regexp munging on the string :

>> YAML.load(YAML.dump(h).gsub(/^(!ruby.*) \? *$/) { |m| "? #{$1}" })
=> {#=>2}

Looks like I’ll be submitting my first bug report to rubylang.

Variables

The two main types of variables in a Makefile are “recursively expanded” (=; equal to) and “simply expanded” (:=; set equal to—thanks Algol).  Recursively expanded is by far the most common and, realistically, the most confusing.  This form of a variable will not perform the substitution until the last possible moment (lazy evaluation), so if you have the line SOURCES = ${CONFIG} demo.c, make will remember that you use ${CONFIG} until it must be known. So if the value of CONFIG changes, the newest version of CONFIG will be used when it is evaluated. Simple expansion occurs when the variable is declared (eager evaluation), if another variable (CFLAGS) is referenced in a simply expanded declaration it will be replaced with the current value of CFLAGS.

Targets

A target occurs on the left-hand side of a rule, typically this is the name of the file you want to build.  There are special cases of this, most commonly, “clean.”  Usually when someone wants to `make clean’ they want all the object files generated from a previous make to be removed.  However, if you were to create a file named `clean’ the rule would never execute because the file clean is up-to-date.  This can be remedied with by a PHONY target.

.PHONY: clean
clean:
    rm -f *.o

This creates a phony target that depends on clean, which tells make to ignore any files named clean.

More on Variables

The important “automatic” variables are talked about in the presentation ($@, $<, $^, $+, and $?).  Also useful is the % expansion variable.  For example, if there was the rule %.o: %.c in a Makefile, this will tell make that to make any file ending in .o will depend on the same file ending in .c (i.e. a rule foo.o: foo.c automatically exists). This will then execute the same commands (say gcc -m32 -Os -o foo.o foo.c) for all files ending in .o. This is a perfect example of why using automatic variables is a great idea.

Well, I think that is everything I have to say about Makefiles in a short amount of time. If you have any questions please feel free to post in the comments. I’ll try my best to answer them promptly!

(While most of the content here is off the top of my head, I did reference the GNU make page. They have everything you wanted to know about make.)

Attachment: Software Development using GNU/Linux (PDF)