Event Counters

If you are unaware, almost every processor manufactured in recent history has some collection of event counters that are incremented when some processor event occurs. These events can range from clock cycles ticking by, instructions being retired, thermal thresholds being passed, or second level cache misses.

So far, I’ve only really used the CPU clock cycles, level 1 cache line replacement, and instructions retired event counters. Your needs might not match mine, so venture over to the Programmer Manual when you need something else!

Event Ratios

Related to the event counters are event ratios. These simple ratios can help you find specific performance issues in your program. For example, if your program does a lot of memory accesses, the processor may need to replaced cache lines frequently. But cache line replacements are naturally occurring in programs, how do we find excessive? Simple! We can just use the ratio of L1 cache replacements to the number of instructions retired. Then we’ll have an idea of how many times per instruction an L1 cache line is replaced.

Using oprofile

First, you’ll have to be running Linux, then you’ll want to install the “oprofile” package. Since this software installs kernel modules for monitoring, you’ll also need root/sudo access to allow the module to be loaded and unloaded for monitoring sessions. Here, I’ll be running as a user and using the sudo command when needed.

opcontrol

opcontrol is main program that lets you interact with the kernel. If you need a down-and-dirty list of the events available for monitoring, opcontrol --list-events will show you all the event counters at your disposal.

On my processor, the default event to monitor is CPU_CLK_UNHALTED which will tell me where the processor spent most of the time executing. If you want to monitor different events, you can specify what event(s) to monitor at the command line.

$ sudo opcontrol --event L1D_REPL:10000 --event INST_RETIRED:10000

The :10000 after each counter simply specifies what the trigger threshold is for raising processor exception. In other words, every 10,000 instructions retired the processor raises an exception that the oprofile daemon will catch and then increment the sample counter for that event. So, if you see that oprofile has 1 sample of the INST_RETIRED counter then the processor has seen 10,000 such events.

Now that we have the event counters configured, we can start the monitoring.

$ sudo opcontrol --start

Since the system is doing other activities, it is best if what you want to monitor can monopolize the system for the while. In my case I build a simple program that purposefully causes the L1 cache to have a lot of misses (l1thrash source code). I’ll also set the program to execute on one processor (CPU 1).

$ taskset 02 ./l1thrash

After it finishes executing, stop oprofile from running and save the profile session on the disk.

$ sudo opcontrol --stop
$ sudo opcontrol --save l1thrash

Now we have our profile saved to disk and we can view it with opreport.

opreport

Finally, we get to see how the program handled! Since we were smart and saved our profile to a session, we’ll have to specify that at the command line. You might want to pipe the output to less since it can be long at times. On my eight core system the output looks ugly.

$ opreport session:l1thrash
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted L1D_REPL events (Cache lines allocated in the L1 data cache) with a unit mask of 0x0f (No unit mask) count 10000
Samples on CPU 0
Samples on CPU 1
Samples on CPU 2
Samples on CPU 3
Samples on CPU 4
Samples on CPU 5
Samples on CPU 6
Samples on CPU 7
    cpu:0|            cpu:1|            cpu:2|            cpu:3|            cpu:4|            cpu:5|            cpu:6|            cpu:7|
  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|
------------------------------------------------------------------------------------------------------------------------------------------------
      541 95.7522      2969  0.9630       301 92.6154       484 69.9422       797 92.6744       707 88.2647       675 90.3614       707 89.8348 vmlinux
        7  1.2389        21  0.0068         6  1.8462         6  0.8671         9  1.0465         6  0.7491         6  0.8032         5  0.6353 oprofile
        6  1.0619         3 9.7e-04         6  1.8462         1  0.1445         3  0.3488         2  0.2497         1  0.1339         4  0.5083 nf_ses_watch
        5  0.8850        16  0.0052         6  1.8462         7  1.0116        25  2.9070        23  2.8714        30  4.0161        27  3.4307 libc-2.5.so
        3  0.5310         2 6.5e-04         1  0.3077         1  0.1445         2  0.2326         4  0.4994         0       0         1  0.1271 libpython2.4.so.1.0
        1  0.1770         0       0         0       0         0       0         0       0         0       0         0       0         0       0 e1000e
        1  0.1770         0       0         0       0         0       0         0       0         0       0         0       0         0       0 irqbalance
        1  0.1770         1 3.2e-04         1  0.3077         0       0         0       0         0       0         0       0         0       0 sshd
        0       0         2 6.5e-04         0       0         0       0         6  0.6977         8  0.9988         8  1.0710        18  2.2872 bash
        0       0         0       0         0       0         0       0         1  0.1163         0       0         0       0         1  0.1271 gawk
        0       0         0       0         0       0         3  0.4335         1  0.1163         2  0.2497         1  0.1339         0       0 bnx2
        0       0         0       0         3  0.9231         0       0         0       0         0       0         0       0         0       0 ehci_hcd
        0       0    305283 99.0179         0       0         0       0         1  0.1163         4  0.4994         1  0.1339         2  0.2541 l1thrash
        0       0        10  0.0032         0       0         0       0        14  1.6279        12  1.4981        19  2.5435        13  1.6518 ld-2.5.so
        0       0         3 9.7e-04         0       0         0       0         1  0.1163         1  0.1248         2  0.2677         2  0.2541 libcrypto.so.0.9.8b
        0       0         0       0         1  0.3077         0       0         0       0         0       0         0       0         0       0 libm-2.5.so
        0       0         0       0         0       0         0       0         0       0         0       0         1  0.1339         0       0 libpthread-2.5.so
        0       0         0       0         0       0         0       0         0       0         0       0         1  0.1339         0       0 syslogd
        0       0         0       0         0       0         0       0         0       0         1  0.1248         0       0         0       0 which
        0       0         0       0         0       0         1  0.1445         0       0         0       0         0       0         0       0 libcups.so.2
        0       0         0       0         0       0         0       0         0       0         0       0         2  0.2677         0       0 libusb-0.1.so.4.4.4
        0       0         0       0         0       0       189 27.3121         0       0        30  3.7453         0       0         7  0.8895 oprofiled
        0       0         1 3.2e-04         0       0         0       0         0       0         1  0.1248         0       0         0       0 cupsd

You may notice that the columns try to be sorted in descending order by the number of samples taken for a specific process. However, on CPU 1 (where we ran l1thrash) the sorted order isn’t close to correct. Luckily, we know that the bulk of our program only ran on CPU 1, so we can reissue the opreport command specifying that we only care about that processor.

$ opreport session:l1thrash cpu:1
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted INST_RETIRED.ANY_P events (number of instructions retired) with a unit mask of 0x00 (No unit mask) count 10000
Counted L1D_REPL events (Cache lines allocated in the L1 data cache) with a unit mask of 0x0f (No unit mask) count 10000
INST_RETIRED:1...|   L1D_REPL:10000|
  samples|      %|  samples|      %|
------------------------------------
  1834500 91.0882    305283 99.0179 l1thrash
   154499  7.6713      2969  0.9630 vmlinux
    21655  1.0752        21  0.0068 oprofile
     2176  0.1080        16  0.0052 libc-2.5.so
      442  0.0219        10  0.0032 ld-2.5.so
      435  0.0216         2 6.5e-04 bash
      108  0.0054         3 9.7e-04 libcrypto.so.0.9.8b
       47  0.0023         3 9.7e-04 nf_ses_watch
       43  0.0021         1 3.2e-04 sshd
       35  0.0017         2 6.5e-04 libpython2.4.so.1.0
       10 5.0e-04         0       0 libavahi-common.so.3.4.3
       10 5.0e-04         1 3.2e-04 cupsd
        9 4.5e-04         0       0 libcups.so.2
        7 3.5e-04         0       0 bnx2
        3 1.5e-04         0       0 libavahi-core.so.4.0.5
        1 5.0e-05         0       0 libpthread-2.5.so
        1 5.0e-05         0       0 timemodule.so

That looks better! Since we’ve narrowed down the output to one CPU, we now get to see both events that we monitored too. You can see that the majority of the time was spent in our l1thrash program, but how did it do?

We know that the number of samples is the number of times that the event counter on the processor hit 10,000 for both counters. So, we find that our l1thrash program caused level 1 cache replacements and retired instructions. Egads! Is that good or bad? Well, now we can throw in our ratio calculation for the L1 data cache miss:

That seems pretty bad to me! We can also see that the Linux kernel (vmlinux) had a ratio of 2,969:154,499 or about 1.9%, that is a fairly typical miss ratio.

A Second Example

This is a real example of a program I am actively trying to improve. The program is a kernel module (nf_ses_watch) designed to intercept packets at a decent rate, it is not performing well. Here I’ll use the default CPU_CLK_UNHALTED event monitor to see where the processor spends most of its time.

$ # I've already loaded the kernel module and started my packet generator
$ sudo opcontrol --event default
$ sudo opcontrol --start
$ # I'll wait about 30 seconds so there are enough samples to be meaningful
$ sudo opcontrol --stop
$ sudo opcontrol --save bombard

Now I have my saved session and can look at the profile. I’ve also taken the time to set the interrupt affinity of the Ethernet device to a specific processor (CPU 7), so now we can see if all the time was spent in my code of Linux code.

$ opreport session:bombard cpu:7
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
  samples|      %|
------------------
   737746 86.6169 nf_ses_watch
    88183 10.3533 vmlinux
    16810  1.9736 e1000e
     3680  0.4321 oprofiled
     2594  0.3046 oprofile
     1578  0.1853 libc-2.5.so
      900  0.1057 bash
       78  0.0092 ld-2.5.so
       52  0.0061 ophelp
       26  0.0031 libavahi-common.so.3.4.3
       22  0.0026 libavahi-core.so.4.0.5
       13  0.0015 gawk
        9  0.0011 libcrypto.so.0.9.8b
        9  0.0011 libpython2.4.so.1.0
        9  0.0011 sshd
        8 9.4e-04 bnx2
        4 4.7e-04 libpthread-2.5.so
        3 3.5e-04 grep
        2 2.3e-04 ipv6
        2 2.3e-04 auditd
        1 1.2e-04 cat
        1 1.2e-04 libdl-2.5.so
        1 1.2e-04 libm-2.5.so
        1 1.2e-04 libpcre.so.0.0.1
        1 1.2e-04 dirname
        1 1.2e-04 automount

Wow! Over 86% of the time we were executing code in the nf_ses_watch kernel module (my code)! Let’s see if we can dig a little deeper. First, oprofile has already done the work for us and tracks the specific symbol name within a piece of code that was active when the sample was taken with the --symbols option (this results in a very long list of kernel symbols). But, in the case of a kernel module, opreport doesn’t know where to find the symbol names so we have to tell it where the kernel module lives with --image-path.

$ opreport session:bombard cpu:7 --symbols --image-path ~/nf_ses_watch/kmod | head
warning: /bnx2 could not be found.
warning: /e1000e could not be found.
warning: /ipv6 could not be found.
warning: /oprofile could not be found.
warning: /sbin/auditd could not be read.
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
warning: could not check that the binary file /home/mjschultz/mon/module/kmod/nf_ses_watch.ko has not been modified since the profile was taken. Results may be inaccurate.
samples  %        image name               app name                 symbol name
733996   86.1767  nf_ses_watch.ko          nf_ses_watch             do_rip_entry
16810     1.9736  e1000e                   e1000e                   (no symbols)
10308     1.2102  vmlinux                  vmlinux                  rb_get_reader_page
9785      1.1488  vmlinux                  vmlinux                  read_hpet
8701      1.0216  vmlinux                  vmlinux                  ring_buffer_consume
3606      0.4234  vmlinux                  vmlinux                  netif_receive_skb
3530      0.4144  vmlinux                  vmlinux                  kfree

(I’ve piped the output through head to keep it reasonable.) We can see the real dirt here! By a huge margin, the do_rip_entry symbol in my nf_ses_watch module executes more than the Ethernet driver that is handling the raw packets. So that is where I’ll be looking when I try to resolve my bug.

Conclusions

If you are looking to optimize your program, oprofile is a great tool to use. The default event monitor (CPU clock cycles on most processors), can give you an idea of what part of your program is using most of the processor time. Once you know that, you can focus your efforts on reducing the number of cycles spent in that function. But don’t forget about all those other events too. If you have a memory intensive application, maybe you could reduce the memory contention and get an effective speedup with almost no refactoring!

(I’ve tried my best to be accurate with this information and I welcome any explicit corrections or clarifications.)

Now that I’m somewhat settled (and a little wiser), I decided to once more try my hand at a solid backup plan.  This was mainly motivated by a recent reinstall of my wife’s system (no lost data, just operating system upgrade).  Since I don’t have vast amounts of time on my hands, I didn’t want to forward port my old solution to get it to work on Linux and Mac OS X, so I looked for new solutions.  I recalled my advisor from Marquette mentioning rdiff-backup as what he put on his wife’s machine during her dissertation days.

As it turns out, rdiff-backup does most of what I wanted out of my backup solution and, in fact, does it a little better.  The main issue I had with my system was that it would periodically (monthly) take a snapshot of my home directory, after that it would periodically (weekly) build incremental diffs based off that snapshot.  What this boils down to is that, if a catastrophic failure happens I would roll back to the most recent snapshot, then progress forward in time to the most recent incremental file.  Not bad, but if you want better-than-weekly granularity it could be a lot of work.  Obviously, I had scripted this part, but still it is wasted time.  With rdiff-backup, it would be a single copy operation to restore to the most recent version.  If you wanted older versions you could roll back through the incremental diffs (again, it is automated).

The other feature that I needed was the ability to remove backups/incremental data older than some time frame (monthly).  Again, rdiff-backup gives me this ability at the command line.  Other bonuses include the fact that it is cross-platform (via macports or most Linux repositories), written in Python, and not maintained by me!

With the basic service in place, it was time to make it automated.  Again, linked off rdiff-backup’s page is an article on how to do unattended backups.  Besides the typical unattended SSH-keypair-without-a-passphrase and protecting-the-account steps, it introduced me to a new trick (which for some reason, despite having the knowledge on how to do it, never put together) using SSH config.

Host athena-backup
	Hostname athena.olympus
	User backups
	IdentityFile ~/.ssh/backups_rsa
	Compression yes
	Protocol 2

Now, if I try to ssh athena-backup, it’ll automatically use the correct identity file and user name which saves me from having to specify it on the command line (which you can’t typically do with wrapper functionality).  More importantly, it doesn’t break normal SSHing onto that host since we made it a special host (that’s the part I never put together, even though I knew it was possible).

The next issue I never took that time to think about before was my having moved from desktop to laptop (thereby making 1:00am backups worthless sense the laptop isn’t always on).  Because rdiff-backup does a roll-back model instead of my roll-forward model, I decided to do hourly backups to my home machine, thus I’ll likely catch at least one of these a day.  But I’m not always at home!  Getting around that is trivial, I’ll just ping the backup server before trying.  If it doesn’t respond, I don’t backup. This is done through:

ping -c1 -t1 $SERVER > /dev/null 2>&1

where $SERVER is just the name of the backup server.  It pings the host once with a timeout of 1 second, if it succeeds the backup continues; otherwise the script exits.

Of course, setting up the cronjob is as simple as:

0 */1 * * * $HOME/.crontab/rdiff-backup.sh

Hopefully this time around the backup solution is more robust than before.

Attachment: rdiff-backup.sh

http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul

It looks like someone likes Ghostbusters at Mozilla.  I just found it amusing.

Now—interesting as the talk was—he suggested everyone read his only non-technical paper titled, “Technology and Courage” [PDF from Sun].  It took me until yesterday to read it, but it was certainly an interesting article.  I recommend everyone read it.

I begin searching online for something that will watch RSS feeds on my behalf and send me an email when it updates.  The first thing I come across is Feed My Inbox, they seem to offer the exact service I want.  Upon closer inspection, they promise to only send one email every 24 hours.  That won’t cut it—I want my updates and I want them now!  After a bit more searching I find rss2email, a simple Python program that keeps track of multiple RSS feeds and converts entries into emails when it executes.  I go through the initial configuration and set up a cronjob to check for new entries every 4 minutes.  Good enough for now.

However, this brings up an annoyance with RSS feeds:

RSS feeds do not provide real-time updates

Once you get down to it, all an RSS feed does is provide some subset of content on a page.  It is still up to the client to ask the server when new content exists.  This fact has bugged me very slightly in the past since I know xkcd updates every Monday, Wednesday, and Friday at 11pm central time, but my RSS subscription in Firefox won’t provide me with the link until (at its discretion) polls the server for new content.  Now, it bothers me slightly more since I know I have to wait at least 4 minutes for my cronjob to run, plus any time it might take Google to publish the updates in the feed (but I’ll ignore that part).

Is there any way to improve this and give end-users real-time updates from content-providers?  It seems like this would be great for users of Twitter and Facebook, since the end-users want to know what is happening now.

One answer seems to be in push notifications, that seem to have been popularized by Blackberry and iPhone applications.  These allow a central server to send a tiny message to the phone that nudges the device that there is data to be had.  Of course, this works very well on phone systems that can easily associate an content generator with a telephone number to contact.  However, it is a bit tougher with IP-only devices that migrate from network to network.  Although, it seems Apple’s Push Notification Service (APNS) should be able to do this.  Either way, this technology seems to be heading in the right direction.

APNS works by maintaining a connection between the client and server, that way when an event happens server side it just sends it on to all the connected clients.  Unfortunately, I’m not convinced at how well these push notifications will scale.  A system implementing IP push notifications seems like it could easily have on the order of 1000s of simultaneous, persistent connections.  According to this “Comparison of Push and Pull Techniques for AJAX,” (tech report, PDF) push-style system do bog down servers a bit.  (Admittedly, the methodology for that paper might not be the best, but I’m guessing the conclusions are valid—I would like to see more/better studies of this.)

This bring me to what I want to see implemented or for someone to point me to the implementation of a distributed content syndication protocol (DCSP).  The high-level view that I think would work (I haven’t thought long or carefully about it), would be similar to other distributed networks.  The content provider would maintain a complete list of current computers subscribed to the feed and the feed itself.  The client would run software that asks the server who to connect to and select a few peers and create a long running connection.  When new content arrives, the server pushes the content to its peers, who push to their peers, and so forth.  This removes the burden of pushing content to all subscribers from the server, giving it scalability (in my mind).  It would then be up to the client to connect and maintain connections with a collection of peers to get the real-time updates.  I’m sure there would have to be some control messages to prevent flooding.  But, it seems like it would give real-time updates to users.

I suppose this would mean I would have to run another program on my system, but it could either be a front-end client that handles my content feeds or a daemon running in the background and set up to deliver an email to a local mailbox (or even a remote mailbox) when fresh content arrives.

Ah well, who knows if it would work.  Hell, maybe I just missed a fact about RSS that doesn’t make it suck as much as I think it does.  Thus ends my stream of though.

Luckily, the solution is fairly straightforward.  Once you have SSH’d into a remote host (using ssh -X), you simply need to run firefox -no-remote. Of course you may want to tack on > /dev/null and an ampersand & to ignore the output and background the task. (Thanks to The Open Sourcer.)

With Firefox 2.x this behaviour was somewhat undocumented, but with Firefox 3.x, running firefox --help from the command line shows the -no-remote option.  It also seems that the default (i.e. -remote), is “documented” on Mozilla’s site for Remote Control of UNIX Mozilla.

If you wanted to make the -no-remote behaviour the default when SSH’d into remote machines, you could simply add a few lines to your bash profile to alias the firefox command.

# If we're forwarding X over SSH, make firefox execute on this machine
if [ -n "$SSH_CONNECTION" -a -n "$DISPLAY" ]; then
    alias firefox='firefox -no-remote'
fi

At least that is what I did.

With mutt I choose to download all my email via POP3 to a local machine where I can read it when I get around to it (rigorous, isn’t it). After I read a message and deem it complete I move it to a folder named after the sender (or possibly a group) where I can grep the files and read them at a later date.

However, after a while these files pile up and I need to periodically compress and archive them. This, of course, gets annoying and frequently forgotten. To solve this I needed a script that could parse messages in a number of mail formats, find a date, and determine if it is beyond some threshold at which point it should be archived. These requirements brought me to archivemail. Archivemail supports several input formats (IMAP, mh, mbox, Maildir), archives the messages, and outputs a single mbox formatted file (that can be compresses). While I’m not a huge fan of the mbox format I can easily deal with it for archived mail.

Archivemail has several perks that fit my requirements quite well. First, it was easy to get (packages availables on OS X, Fedora, Debian, and Ubuntu), this probably stems from the fact that it is written in python and can easily run on almost any system. Next, it provides several useful command line options, I personally have a cronjob that archives four message folders every 30 days (logwatches and mail lists) and archives other messages after 180 days. This is simply done with the --days command line switch. I also specify a directory to dump all the archived messages into so they don’t clutter up my mail directory. Depending on how you handle you mail there are also options to not archive unread messages or only archive messages older than some fixed date.

For those interested, here is my script that I run as a weekly cronjob to archive and compress my mail messages:

ARCMAIL="/usr/bin/archivemail --quiet --output-dir=$HOME/mail/archive/"

$ARCMAIL --days  30 $HOME/mail/logwatch \
                    $HOME/mail/netflix  \
                    $HOME/mail/amazon   \
                    $HOME/mail/dreamhost

$ARCMAIL --days 180 $HOME/mail/*

Fairly straightforward, eh?

To search through an archive you can just change into the archive/ directory and execute a gunzip -c <filename> | grep <word>. Alternatively, you can use mutt’s built in search and run gunzip <filename>.gz ; mutt -f <filename>.

So, I’ve got Bazaar down pretty well—it’s design has been engineered for a smooth transition from Subversion—but Git is a different beast.

Git breaks a lot of the conceptual assumptions made in svn-land that you didn’t even know you were using to understand your daily VCS use; the basic “checkout, update, commit” operations don’t cleanly map to the git paradigm. This can create many WTF moments spent staring at lines of help like:

git-rebase - Forward-port local commits to the updated upstream head

The payoff for diving into these treacherous waters is, IMHO, worth it. When your first ‘git commit’ or ‘git checkout’ comes back within a few milliseconds and you’re left sitting there still trying to believe it’s already done… this is when you realize “Wow, this tool might actually change the way I work.”

  OPS = {
    "="  =>  lambda { |v, r| v == r },
    "!=" =>  lambda { |v, r| v != r },
    ">"  =>  lambda { |v, r| v > r },
    "<"  =>  lambda { |v, r| v < r },
    ">=" =>  lambda { |v, r| v >= r },
    "<=" =>  lambda { |v, r| v <= r },
    "~>" =>  lambda { |v, r| v = v.release; v >= r && v < r.bump }
  }

Found deep in the rubygems source.

First of all, the reason I was using YAML was as a quick, simple, human-readable backend for a tiny little side-project. I chose YAML over SQLite or CSV because neither have the human-readable thing going for them, plus I’m working in Ruby and it seemed to have nice built-in support for YAML.

You can take any object you like (with obvious exceptions like streams and sockets) and just YAML.dump it, creating a happy string waiting to be written to the nearest file. So given this class:

class Foo
  def initialize
    @bar = 'one'
    @baz = 2
    @qux = [3, 'four', 5]
  end
end

You can do something like this:

>> f = Foo.new
=> #
>> YAML.dump(f)
=> "--- !ruby/object:Foo \nbar: one\nbaz: 2\nqux: \n- 3\n- four\n- 5\n"

And that YAML string at the end there looks like this:

--- !ruby/object:Foo
bar: one
baz: 2
qux:
- 3
- four
- 5

So you can stuff that wherever you’d like (file.yml let’s say) and then at some later date all you need to do is:

>> new_f = YAML.load_file('file.yml')
=> #
>> f == new_f
=> true

Fun times, no? And most importantly fun times that don’t require one learns YAML in great detail. Well, let me show you exactly where the fun times ended for me. For an example, let’s make a simple hash with our f.

>> h = { f => 2 }
=> {#=>2}
>> YAML.dump(h)
=> "--- \n!ruby/object:Foo ? \n  bar: one\n  baz: 2\n  qux: \n  - 3\n  - four\n  - 5\n: 2\n\n"

Seems fine, no?

---
!ruby/object:Foo ?
  bar: one
  baz: 2
  qux:
  - 3
  - four
  - 5
: 2

So we go about our business, right?

>> YAML.load(YAML.dump(h))
ArgumentError: syntax error on line 2, col -1

Boo. So now I have to go and do what I was trying to avoid doing in the first place: learn about YAML. Well after wading through the schema documents for YAML, which are long, full of BNF, and no fun. At the end of the day it turns out that YAML.dump produces invalid YAML whenever an object is used as a key in a Hash. Without going into too much detail (partially because I don’t fully understand it), the question mark needs to come before the ruby object tag for it to be valid.

Here’s a workaround that just does some regexp munging on the string :

>> YAML.load(YAML.dump(h).gsub(/^(!ruby.*) \? *$/) { |m| "? #{$1}" })
=> {#=>2}

Looks like I’ll be submitting my first bug report to rubylang.