international geographic

Sounds easy, doesn’t it? Well it actually is quite simple but the error messages along the way can really trip you up!

My first attempt at profiling a bit of code was to use the full-boat Eclipse stack: Eclipse Test & Performance Tools Platform Project! Well, what they don’t tell you anywhere on the project page is that it’s only supported on Windows and Linux. A Mac port was started sometime around 2004 and never completed. Yeah, it’s been that long!

And so this brings us to Apple’s Shark: an extremely barebones, no-frills profiler, but, what the heck, it’s free. For the basics on using this tool with your Java app, this great post has all the details. There’s just one catch: 64-bit support. There is none. If you’re on a 64-bit stack and you try to run it, you’ll see something like the following:

$ java -agentlib:Shark -cp foo.jar com.foo.Bar
Error occurred during initialization of VM
Could not find agent library: libShark.jnilib (searched /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Libraries:/System/Library/Java/Extensions:/Library/Java/Extensions:.)

Er, what? Now let’s see here…

$ ls -al /System/Library/Java/Extensions/libShark.jnilib 
-rwxr-xr-x 1 root wheel 50352 Oct 24  2010 /System/Library/Java/Extensions/libShark.jnilib

Well, that’s odd. But it said it looked there, right?! Well, as it turns out, the Shark JNI library only supports 32-bit JVMs. So finally, we arrive at the following:

$ java -d32 -agentlib:Shark -cp foo.jar com.foo.Bar
2011-05-18 23:05:20.756 java[84473:20f] Shark for Java enabled...

Great sucess! And now, back to work, just a little less sane…

I recently compiled Q4M on my Mac and I’m documenting it here now because it proved to be somewhat painful so hopefully you’ll avoid a bit of grief by reading this.

First off, you need MySQL Server. I’ve been using MacPorts for some time now, so this is installed via ports. Unfortunately, it seems to be a pretty barebones config so I had to create a custom variant to add the necessary --with-fast-mutexes option. You can grab my modified Portfile from github. Just clone it into some directory and then add that path into /opt/local/etc/macports/sources.conf before the rsync line. Once you have that, go ahead and reinstall MySQL:

$ sudo port install mysql5 +fast_mutexes

Q4M also requires access to the MySQL sources, so let’s get those unpacked:

$ cd /tmp
$ tar -xzf /opt/local/var/macports/distfiles/mysql5/mysql-5.1.57.tar.gz

Once that’s done, follow these steps to get Q4M compiled:

$ cd /opt/local/include
$ sudo ln -s mysql5/mysql 
 
$ ./configure --with-mysql=/tmp/mysql-5.1.57 \
    --prefix=/opt/local/
 
$ CPATH=/opt/local/include/mysql \
   LIBRARY_PATH=/opt/local/lib/mysql5/mysql \
   make
 
$ sudo mv /opt/local/lib/libqueue_engine.* \
   /opt/local/lib/mysql5/mysql/plugin/
 
# Final step, tell mysql to install the engine
$ mysql -uroot < support-files/install.sql

If after all that you’re still having problems, maybe you’ll have better luck with these Japanese instructions.

I have no idea how Gabriel ever gets anything done, considering that he’s constantly responding to emails, tweets, forum posts, comments on various sites and IRC — just to name a few!

Here’s a typical exchange from earlier today when I reported a poorly performing query on DDG:

I used to be like that when I was at Operative but I don’t think I was anywhere near as productive as him.

I spent a bunch of time the last couple of days sprucing up a few of my personal projects. Today was mostly spent on the lunchomizer, which gives you random suggestions for lunch, based on your location. It still only works for Midtown Manhattan since the locations all come from Midtown Lunch and that’s also the neighborhood where I happen to work.

The biggest change today was fixing the HTML5 location support and a much better mobile experience via a separate Rails layout tailored for small screens. All in all, it was pretty easy, but one thing that surprised me was a noticeable lack of a de facto standard library for doing mobile, or really any, browser detection. I ended up using a gem called Nomadic and that worked well with the following controller code:

class LunchController < ApplicationController
    layout :choose_layout
    [...]
    private
 
    # use mobile layout for mobile clients (using browser detection)
    def choose_layout
        Nomadic.mobile?(request.env) ? "mobile" : "default"
    end
end

A fairly simplistic solution, but perfect for this app which has only a single controller and a single view :)

And for the curious, the full source is available on github. Fork away!

I got this creepy voicemail today on an unused Vonage number that I have:

Vonage transcription: “Good morning and good morning and good more and then I’m. I’m alone. So in the shower. I am gonna be more than any card board boat races in my dreams. Swimming in the lake trip the water”

PS. this is just an excuse for me try out the new HTML5 <audio> tag, which apparently doesn’t work in Firefox 3.6.x.

So it turns out that Windows Firewall, even in Windows 2008, still can’t accept a range of ports, either in the UI or via command line, most commonly when setting up PASV FTP transfers. The common workaround is to create one entry per port in your range like so:

C:> FOR /L %I IN (60000,1,60200) DO netsh firewall add portopening TCP %I "Passive FTP"%I

While this does work, it’s slightly annoying that you have to create 200 individual entries in your config. My slightly better workaround is to just stick every port into the text entry field using this simple ruby helper:

puts (60000..60200).to_a.join(",")

You can run this either in IRB or directly on the command line using the following command:

$ ruby -e 'puts (60000..60200).to_a.join(",")'

This is great if your workstation has ruby on it, but sucks otherwise. So use this javascript version right here instead!

Start:

End:

Range:

Oh boy, here we go again. I actually managed to come up with a list of my top 3 this year, so let’s just cut to the chase, shall we?

1. Kanye West – My Beautiful Dark Twisted Fantasy
2. LCD Soundsystem – This is Happening
3. Titus Andronicus – The Monitor

And the runner-ups in alphabetical order:

• Arcade Fire – The Suburbs
• Beach House – Teen Dream
• Best Coast – Crazy For You
• Das Racist – Shut Up, Dude
• Das Racist – Sit Down, Man
• Robyn – Body Talk
• The-Dream – Love King
• Toro Y Moi – Causers Of This
• Wavves – King of the Beach
• Wild Nothing – Gemini
• Xiu Xiu – Dear God, I Hate Myself

Best EPs:

• Annuals – Sweet Sister EP
• DEERPEOPLE – DEERPEOPLE EP
• Girls – Broken Dreams Club EP

Can’t wait for next year!

After spending the better part of a day trying to get HDFS to mount on my Mac, I finally gave up. Luckily, I was able to find MuCommander, a cross-platform Java port of the Norton Commander of old, and as luck would have it, it supports HDFS in the latest version! Very handy for quickly browsing your HDFS clusters if you can’t mount it or don’t have the Hadoop toolset installed.

Hadoop has a built-in feature for easily distributing JARs to your worker nodes via HDFS but, unfortunately, it’s broken. There’s a couple of tickets open with a patch again 0.18 and 0.21 (trunk) but for some reason they still haven’t been committed yet. We’re currently running 0.20 so the patch does me no good anyway. So here’s my simple solution:

I essentially copied the technique used by ToolRunner when you pass a “libjars” argument on the command line. You simply pass the function the HDFS paths to the JAR files you want included and it’ll take care of the rest.

Example usage:

public int run(String[] args) throws Exception {
 
    JobConf job = new JobConf(getConf());
 
    // ... job setup ...
 
    NerfUtils.addJarsToJobClasspath(job, 
        new String[] { 
            "/libraries/java/solr-commons-csv-1.4.1.jar" });
 
    // ... more job setup ...
 
    return JobClient.runJob(job).getJobState();
}

Might not be the prettiest or best solution but it works for me!

Using Hadoop’s DistributedCache mechanism is fairly straightforward, but as I’m finding is common with everything-Hadoop, not very well documented.

Adding files

When setting up your Job configuration:

// Create symlinks in the job's working directory using the link name 
// provided below
DistributedCache.createSymlink(conf);
 
// Add a file to the cache. It must already exist on HDFS. The text
// after the hash is the link name.
DistributedCache.addCacheFile(
    new URI("hdfs://localhost:9000/foo/bar/baz.txt#baz.txt"), conf);

Accessing files

Now that we’ve cached our file, let’s access it:

// Direct access by name
File baz = new File("baz.txt");
// prints "true" since the file was found in the working directory
System.out.println(baz.exists()); 
 
 
// We can also get a list of all cached files
Path[] cached = DistributedCache.getLocalCacheFiles(conf);
for (int i = 0; i < cached.length; i++) {
    Path path = cached[i];
    String filename = path.toString();
}