Very Low Defect Project

When people talk about Extreme Programming, they often focus on such things as its adaptive planning style, or its evolutionary approach design. One small but growing trend that particularly interests me is the small but growing number of XP projects that have very low defect rates, by which I mean less than one production bug per month.

The first of these I ran into was Kent's description of a firm that makes food sorting machines. These things have conveyor belts that run past cameras and other sensors which will sort fruit and vegetables into various categories (all this stuff driven by Smalltalk). The team had a hundred or so open bugs at one time, but since adopting XP the bug rate dropped to where they were getting a bug roughly every couple of months.

I was particularly happy to run into the second case I saw - since it involved several of my old friends from C3. They are now building some portal software at Chrysler. Although they started off at the shocking defect rate of one a month, during 2002 they recorded exactly one bug against their system. During this time they were releasing new version of their software every week or two.

One software company has made a big shift to XP. A lot of its work has involved large legacy systems, which are a challenge. But in some new developments they have had some very low defect rates. Two such projects are running at under a bug a month. One of these found pre-install government certification reduced from weeks to days.

We are also seeing some of this at Thoughtworks. It's early since release on our candidate very low defect projects, but a few are seeing similarly low bug rates.

Although these low rates are only achieved by a minority of projects, it does seem to be a common result that teams that do a serious adoption of XP see significantly lower bug rates. I've heard many tales of significant drops in defect rates, even if they didn't get down to sub-bug-month levels.

I need to stress that so far only a few XP teams are getting the very low bug rates. I have only run into a handful so far. The teams seem to be pretty disciplined and led by people who have a year or two of XP under their belt.

Since this is still a small proportion of projects you should not assume you are going to get super-low bug rates by just adopting XP. Nor should you assume that other processes can't get these kinds of bug rates as well.

But what I can conclude is that these kinds of bug rates are achievable in software development. I can also conclude that some teams who get there consider XP to be a very important tool in helping to do it.