According to the official blog post, the new and improved Delicious brings us speed, usability, and oh so good looks among other features and it was a long time in the making. The Yahoo acquisition was announced on Dec 9, 2005 and the new site finally went live a little over two and a half years later on July 31, 2008. So why did it take them so long?

A key change as a result of the Yahoo! acquisition was their decision to rewrite the whole thing in PHP using the Symfony framework, for no other reason than that it’s the current corporate standard at Yahoo!. Oh, and, coincidentally, Yahoo! Bookmarks was also built on PHP+Symfony.

So now it starts to make a bit of sense: you take a system being actively used by millions of users around the world and you start over from scratch with the goal of building it bigger and better, toss in a couple of hot buzzwords to meet Web 2.0 compliance guidelines, and before you know it 2 years have gone by.

I find it very hard to believe that with all the talent and the thousands of man years combined software development experience over there, that no one understands the pros and cons of rewriting vs refactoring a code base, especially given the enormous success of the service and the relatively trouble-free history as compared to, say, Twitter.

At the same time, I understand it all too well. From where I sit, and having been involved in a similar situation in the past as well as with my current employer, the decision to move to PHP was clearly not based on a cost/benefit analysis of maintaing the current system. In fact, I wonder if they even understood what the real problems, if any, were with the existing system before deciding to not just rewrite it, but write it in another language.

Moving to another language is a pretty drastic step to take and will rarely solve your problem.

Lately I’ve been working on optimizing our stored procedure library which is primarily responsible for generating all sorts of fancy reports for the users. We use lots of nested procedure calls and finding potential targets for optimization is a tricky and time consuming job. Enter Ghetto Profile.

It’s still pretty basic but gets the job done. How’s it work?

1
2
3

ghetto_profile.rb --attach crappy_code.sql
mysql < crappy_code.sql
ghetto_profile.rb --stats -uroot -pmysql

The attach command simply wraps all stored procedure calls with debug statements which can later be analyzed using the stats command.

CALL debug.ON('stored_proc1'); -- added by ghetto_profile
    call stored_proc1(...params...);
CALL debug.off('stored_proc1'); -- added by ghetto_profile

And the output of the stats command is something like this:

total time: 3718.3220
 
stored_proc1
         pct:   75.3873
         total: 2803.1430 (46.72)
         avg:   0.0895
         count: 31327
         min:   0.0860000000000001
         max:   0.343
 
stored_proc2
         pct:   22.9479
         total: 853.2760 (14.22)
         avg:   0.0273
         count: 31240
         min:   0.0
         max:   0.371
 
stored_proc3
         pct:   1.1224
         total: 41.7340 (0.70)
         avg:   0.0007
         count: 59020
         min:   0.0
         max:   0.074

Wow, that stored_proc1 is taking up 75% of the time! Hacking ensues…

total time: 1151.3400
 
stored_proc1
         pct:   0.4372
         total: 5.0340 (0.08)
         avg:   0.0001
         count: 43892
         min:   0.0
         max:   0.007

Totally sweet! Ok, enough talk. Get some:

Ghetto Profile 0.7 (requires ruby, BSD licsensed)
debug.tgz (helper package to setup the debug database and procs)

• March 25 - Crystal Castles with Health (Studio B)
• April 8 - Explosions in the Sky (Terminal 5)
• May 5 - The Teenagers (Bowery Ballroom)
• May 18 - Black Kids (Bowery Ballroom)

Fortunately, that all appears to be changing. I was glad to learn that the recently announced Live Documents is made entirely by an Indian company based out of Bangalore (though it was founded by Sabeer Bahtia of Hotmail fame). I’d be curious how their product does in the Indian market, as opposed to the rest of the world, for which it seems to be primarily targeted.

In this regard management is also to blame, especially when it comes to dysfunctional schedules, wrong incentives, poor hiring, and demoralizing policies.

So how do we learn?

Let’s suppose that you wanted to learn how to survive a volcanic eruption. You have two basic options: go and find someone who’s done it before (Tommy Lee Jones?) and learn directly from them, or, find a book or other information source and learn it on your own.

Where I work, both options are impractical, for different reasons, within our product, development and quality assurance groups. As part of the dev team, I’ll only speak to what I’ve observed here but I think it applies just as well to the other groups.

Learning through the sharing of experience

In our core team, we currently have 31 developers (excluding myself): 5 leads, 7 senior developers, and 19 junior. Ranks are strictly based on years of experience, which has no bearing at all on quality of experience. In essence, the knowledge gap between our junior and senior developers is small, and even smaller between our seniors and our leads. After a certain amount of time in the team, a developer sort of plateaus — this is how we do things, and, well, that’s that. The mentality is, “this is how I was taught by the guy before me and it seems to work well enough, so why change?”

Self learning

We’ve got some bright people on the team, there’s no question of that. If given the opportunity, they’re capable of doing good work. The problem we face is that they just haven’t been given the opportunity. Not once, in three years. Our schedules are the most aggressive I’ve ever seen, without any break between releases. The schedule simply does not allow for any amount of experimentation with new techniques or technologies. We tried introducing some change in to one of our current projects and we’ve been burned pretty badly by it; we slipped on a date for the first time since I’ve been around, and the repercussions have been swift in the making. More on that later, but first…

Some history

The software division started out as many software stories do: a couple of smart people with lots of domain knowledge, but no technical expertise, and no product, went out and sold an idea. Through sheer force of will, and the help of one very sharp developer, they managed to build and deliver on their promises in a matter of months. As the deals rolled in, and with them, requests for more features, the engineering team grew slowly but things quickly boiled over, causing that first developer, who started it all, to leave abruptly [1].

From there, it was a long, slow descent to the present day. The developer wasn’t replaced quickly and the decision was made to move core development offshore, at a much lower cost. No one back at headquarters really understood what was going on in development, just that things seemed to be working smoothly. Dates were hit, features were delivered, customers were happy, and their numbers were growing. What more could you want?

Well, at some point it was decided that an entirely new platform was required for the company to continue on it’s current growth path. A “forklift” project that would be developed in parallel and eventually replace the existing system, which would continue to be developed as demand was still strong and new customers were being signed on. New talent was finally brought in but the existing resources never benefited from them, as they worked in isolation. The goals for the new project were technologically lofty, to put it mildly. Time passed and milestones were missed, dates slipped further and further, and eventually the project was scrapped in favor of the existing platform which continued to enjoy success, in that it was always on schedule.

Everybody panic

Cut to the present. We’ve had a few production outages, with some lasting for more than a single business day. The type of business we’re in, and the money at stake, not to mention the promises we’ve made, this is totally unacceptable. We’ve had dates slip. The platform we’re building on top of is unstable to the point where every release only compounds the performance problems we see in production. Our customers aren’t stupid — they’ve noticed the consistent performance degradation and have been complaining pretty steadily, so far to deaf ears.

The reaction? Knee jerk. I’m told a consultant is being brought in, and a project manager hired and set to start in a week’s time. I’ve no idea how they came to this, as no one in my team, now numbering 42 developers at last count, was interviewed or consulted before any decision was made. In fact, I’m not even sure my boss was.

I think it might be a small step in the general vicinity of something vaguely trying to resemble the appearance of being in the right direction, but, given past performance, I don’t think it’s going to work. There is a tendency to stop short of solving the real problems, and in doing so, generally causing more harm than good. I’ve some thoughts on a real solution, but I’ll leave that to another post.

[1] “Sure, I’ve been there, getting the wrong answers from the boss. Some business people just aren’t good at making informed decisions even when you’ve patiently explained all the tradeoffs and benefits. I think we, as programmers, need to be sure we’re holding up our end of the bargain - delivering all the data the business-people need to make good decisions. When they make bad ones anyway then it may be time to find a new job. Until then it’s our job to follow orders, and to make the best use of the time we’re given.”

Without a commitment from all the concerned parties, how can one expect any enterprise to be successful? Be it business, government, military or otherwise.

My group has shown it’s commitment time and again, but we haven’t seen the same from others. How do you find the motivation to stay committed in a situation like that?

I think I’ll just go to sleep instead.

It wasn’t until they put a signal at the [fairly minor] intersection near the office that I realized just how well “organized” this particular brand of chaos can work. Right away, about five minutes in the morning and ten minutes in the evening were added to my commute, just sitting in traffic. The very next day, the light changed to a flashing yellow (go slow!) with a traffic cop directing traffic. In the evening it seemed to be working normally but traffic was much more fluid. Ten minutes sitting still became roughly 5 minutes. Either people took a different route tonight or they just got used to the fact that, sometimes, you need to stop at an intersection.

Strange Things with Bacon