Here’s a typical exchange from earlier today when I reported a poorly performing query on DDG:

I used to be like that when I was at Operative but I don’t think I was anywhere near as productive as him.

According to the official blog post, the new and improved Delicious brings us speed, usability, and oh so good looks among other features and it was a long time in the making. The Yahoo acquisition was announced on Dec 9, 2005 and the new site finally went live a little over two and a half years later on July 31, 2008. So why did it take them so long?

A key change as a result of the Yahoo! acquisition was their decision to rewrite the whole thing in PHP using the Symfony framework, for no other reason than that it’s the current corporate standard at Yahoo!. Oh, and, coincidentally, Yahoo! Bookmarks was also built on PHP+Symfony.

So now it starts to make a bit of sense: you take a system being actively used by millions of users around the world and you start over from scratch with the goal of building it bigger and better, toss in a couple of hot buzzwords to meet Web 2.0 compliance guidelines, and before you know it 2 years have gone by.

I find it very hard to believe that with all the talent and the thousands of man years combined software development experience over there, that no one understands the pros and cons of rewriting vs refactoring a code base, especially given the enormous success of the service and the relatively trouble-free history as compared to, say, Twitter.

At the same time, I understand it all too well. From where I sit, and having been involved in a similar situation in the past as well as with my current employer, the decision to move to PHP was clearly not based on a cost/benefit analysis of maintaing the current system. In fact, I wonder if they even understood what the real problems, if any, were with the existing system before deciding to not just rewrite it, but write it in another language.

Moving to another language is a pretty drastic step to take and will rarely solve your problem.

So how do we learn?

Let’s suppose that you wanted to learn how to survive a volcanic eruption. You have two basic options: go and find someone who’s done it before (Tommy Lee Jones?) and learn directly from them, or, find a book or other information source and learn it on your own.

Where I work, both options are impractical, for different reasons, within our product, development and quality assurance groups. As part of the dev team, I’ll only speak to what I’ve observed here but I think it applies just as well to the other groups.

Learning through the sharing of experience

In our core team, we currently have 31 developers (excluding myself): 5 leads, 7 senior developers, and 19 junior. Ranks are strictly based on years of experience, which has no bearing at all on quality of experience. In essence, the knowledge gap between our junior and senior developers is small, and even smaller between our seniors and our leads. After a certain amount of time in the team, a developer sort of plateaus — this is how we do things, and, well, that’s that. The mentality is, “this is how I was taught by the guy before me and it seems to work well enough, so why change?”

Self learning

We’ve got some bright people on the team, there’s no question of that. If given the opportunity, they’re capable of doing good work. The problem we face is that they just haven’t been given the opportunity. Not once, in three years. Our schedules are the most aggressive I’ve ever seen, without any break between releases. The schedule simply does not allow for any amount of experimentation with new techniques or technologies. We tried introducing some change in to one of our current projects and we’ve been burned pretty badly by it; we slipped on a date for the first time since I’ve been around, and the repercussions have been swift in the making. More on that later, but first…

Some history

The software division started out as many software stories do: a couple of smart people with lots of domain knowledge, but no technical expertise, and no product, went out and sold an idea. Through sheer force of will, and the help of one very sharp developer, they managed to build and deliver on their promises in a matter of months. As the deals rolled in, and with them, requests for more features, the engineering team grew slowly but things quickly boiled over, causing that first developer, who started it all, to leave abruptly [1].

From there, it was a long, slow descent to the present day. The developer wasn’t replaced quickly and the decision was made to move core development offshore, at a much lower cost. No one back at headquarters really understood what was going on in development, just that things seemed to be working smoothly. Dates were hit, features were delivered, customers were happy, and their numbers were growing. What more could you want?

Well, at some point it was decided that an entirely new platform was required for the company to continue on it’s current growth path. A “forklift” project that would be developed in parallel and eventually replace the existing system, which would continue to be developed as demand was still strong and new customers were being signed on. New talent was finally brought in but the existing resources never benefited from them, as they worked in isolation. The goals for the new project were technologically lofty, to put it mildly. Time passed and milestones were missed, dates slipped further and further, and eventually the project was scrapped in favor of the existing platform which continued to enjoy success, in that it was always on schedule.

Everybody panic

Cut to the present. We’ve had a few production outages, with some lasting for more than a single business day. The type of business we’re in, and the money at stake, not to mention the promises we’ve made, this is totally unacceptable. We’ve had dates slip. The platform we’re building on top of is unstable to the point where every release only compounds the performance problems we see in production. Our customers aren’t stupid — they’ve noticed the consistent performance degradation and have been complaining pretty steadily, so far to deaf ears.

The reaction? Knee jerk. I’m told a consultant is being brought in, and a project manager hired and set to start in a week’s time. I’ve no idea how they came to this, as no one in my team, now numbering 42 developers at last count, was interviewed or consulted before any decision was made. In fact, I’m not even sure my boss was.

I think it might be a small step in the general vicinity of something vaguely trying to resemble the appearance of being in the right direction, but, given past performance, I don’t think it’s going to work. There is a tendency to stop short of solving the real problems, and in doing so, generally causing more harm than good. I’ve some thoughts on a real solution, but I’ll leave that to another post.

[1] “Sure, I’ve been there, getting the wrong answers from the boss. Some business people just aren’t good at making informed decisions even when you’ve patiently explained all the tradeoffs and benefits. I think we, as programmers, need to be sure we’re holding up our end of the bargain – delivering all the data the business-people need to make good decisions. When they make bad ones anyway then it may be time to find a new job. Until then it’s our job to follow orders, and to make the best use of the time we’re given.”

For me it’s almost an out of body experience where I look [down] at myself in a particular instant and I try to work backwards to the events that led up to it. It seems to be happening a lot, of late, and it always comes back to the same question. Two weeks ago, I was sitting in the domestic terminal at the Bombay airport, reading a book of no consequence and listening to my iPod while waiting for my flight to Pune when I suddenly looked up and thought to myself, “what the fuck am I doing here? How the hell did I get here?”

It happened again yesterday. I was sitting in my hotel room, eating a fabulous lunch (brunch, really, but it was 3pm; scrambled eggs, french toast, hash browns, chicken sausage, and hot chocolate) and looking out the window, out over the gloomy sprawl of Chennai before me, when it came up again: “what the fuck am I doing here? How the hell did I get here?”

There’s an obvious answer (I’m here for Operative as their resident software ninja, blah, blah) but that’s not good enough. It doesn’t really answer the question as to why I’m here, in the here and now. Even more, I can think of lots of reasons that I shouldn’t be here; why I should be somewhere else, doing something else. And only a few reasons (e.g., conscious decisions I’ve made) for being here.

And yet here I am.

It took me a really long time to answer the question, because to me it doesn’t affect how I judge a person’s performance, and it was difficult for me to look at the question objectively and realize that not everyone thinks the way I do. Everyone has their own criteria for judging performance or value. Which got me thinking about meritocracies and why they don’t work in business.

Many large, successful Open Source projects are, or appear to be (I’ll get to this later), meritocracies. The Apache Software Foundation (ASF) is a popular example. So what’s so different about open source software projects that a meritocracy not only works but seems to come so naturally?

Clearly much of it has to do with the voluntary nature of open source; when faced with an adverse situation in the management of a project you have two clear options when all else fails: stop contributing or fork. Both options carry an equally low risk, from an individual standpoint. The same two options are available in the business world, however the risks are much greater when one’s career and livelihood lie in the balance but the rewards are potentially much greater for forking — e.g., quit and start a competing product or service.

Even in the latter case, can a meritocracy really work in the long run? The incubation phase of almost any startup is entirely merit-based — from acquiring funding to competing in the marketplace, both will be judged on the merits of the ideas and the products or services the business is able to produce. But how long can this be sustained? Once a company grows out of it’s core group of people into a full-fledged business, it becomes increasingly difficult to keep the same level of excellence and far more costly to replace people at the upper levels. The stakes are also much higher for everyone involved, as noted by the ASF:

“What is interesting to note is that the process scaled very well without creating friction, because unlike in other situations where power is a scarce and conservative resource, in the apache group newcomers were seen as volunteers that wanted to help, rather than people that wanted to steal a position.”

So it becomes an issue of power and the ability to wield it. Bill Gates commented on a recent survey, saying: “Communication skills and the ability to work well with different types of people are very important too,” he said. “Software innovation, like almost every other kind of innovation, requires the ability to collaborate and share ideas with other people, and to sit down and talk with customers and get their feedback and understand their needs.” In other words, to innovate you need more than just technical ability and great ideas. You need the ability to communicate those ideas and work with people who may not always be as strong technically.

And that’s where strong leadership comes in. The problem with any large organization is that it’s almost impossible for everyone to be in complete agreement all the time. Even in a merit based organization like the ASF, consensus is not a given and people have found ways to game the system. Without strong leadership at the top you’ll have a constant power struggle to fill that void at every level in the organization, as different groups vie for control. (See also: IRAQ)

The best ideas can win out, but sometimes it takes a dictator.