Blog

Clients who are new to software development often have a hard time accepting the uncertainty that's inherent in the work. They frequently have experience in other industries where the work can be estimated and completed confidently within a narrow range. They expect the same from software projects and are frustrated when their expectations are not met. Past experience has taught them that a group of craftsmen with similar skills and ability can complete the same well-specified task in about the same amount of time. They reasonably ask, for a well-specified software task, there ought to be a "correct" estimate... right?

The answer is "no, not really".

Even for small, clearly defined tasks, different programmers will take different amounts of time to complete them. This is true even if the programmers have similar skills and experience. There are many ways to explain this: Programming is more art than science. The problems are complex. The solutions are non-linear. There are many different ways to accomplish the same thing. Etc. But normally, these explanations fall on deaf ears. If you haven't written software yourself, it's hard to grasp the empirical reality. You may think "OK, I can understand that there are lots of ways to solve a problem in software, but my plumber tells me the same thing when he comes to unclog my drain..."

Instead of casting about for better explanations, I'll describe some real world data. At Art & Logic, the cornerstone of our recruiting process is a programming test - the Art & Logic Programming Challenge (ALPC). We ask all candidates to complete it. It contains several, small, well-specified programming tasks. Candidates complete the tasks then send us their solutions and the time they spent on them.

I collated the results for a subset of candidates: developers we've hired over the past several years. This assures that the data points come from programmers with similar abilities. We don't want to skew the results by including developers who were not up to A&L's standards.

The results are interesting. Here are histograms for two parts of the ALPC:

Both charts resemble normal distributions (i.e. "bell curves") with the addition of one or two outliers. In the first chart, the mean is 3.8 hours. There are two outliers at 2x (where x is the mean) and one at nearly 3x. In the second chart, the mean is 4.4 hours. The outlier is almost 3 times that.

The other thing that's interesting to note is the variability in the normal range. Both charts fall roughly in the range of 4 +/- 2 hours. That's a 300% difference between the low end and the high end.

So what's the take away here?

It's normal for different programmers to take different amounts of time to complete the same tasks. Usually, results will fall within a predictable range, albeit a broad one. Occasionally, a programmer will take dramatically longer. This is normal. It doesn't mean that the programmer has done anything wrong.

If your company is involved in software development, what I have just described as "normal"poses significant challenges to you. How do you confidently prepare a budget and a schedule for your project? How do you keep your project on track? If even good programmers sometimes take much longer to complete things, how do you know if your programmer is good?

The first step in facing these challenges is accepting the reality of the situation. There are many factors that you can influence in a software project. The uncertainty inherent in the work itself is not one of them.