Anecdotal Evidence

One of the frustrations of the software development field it's hard to choose between different techniques and tools. Often when someone talks about this they are asked for 'hard data' that the technique or tool is better than alternatives. It's an understandable request, but in the end it's a doomed one. For a start we CannotMeasureProductivity.

So failing hard data, we regularly resort to anecdotal evidence. Indeed my whole career is about spreading ideas based on the analysis of anecdotal evidence. Despite its inferiority to objectively measured phenomena, its unwise to dismiss it. After all, how else can we learn? We learn a lot from our own experiences, but when others tell us their's it adds a lot to our information source.

This is why I'm so keen on seeing people report their experiences, even if they are particular and not backed up by measurements. Readers understand these limitations and will take what they can if they can apply these lessons to their own circumstances.

Last year I was involved on the program committee for a conference and reviewed three papers, all of which followed the same basic theme. Each one discussed an idea that could be used to improve software development. The crucial flaw in each of them was that the authors had not tried their idea out - even once. That's why I voted for rejection on all of them.

Some people would extend this - saying you can't really talk about an idea until you've seen it on multiple projects. While this is nice, I don't agree that it's necessary. Just reporting on one particular thing you discovered on a project is useful because it provides raw material for others. Someone else might be in a similar position and your idea gives them something to try. Someone else may have done a similar thing and when they write about their experiences they report theirs on top of yours.

This approach - some people report their story, others copy it with or without success - is the backbone of how a profession can learn. The fact that it's anecdotal doesn't stop it from working - after all there is no scientifically rigorous evidence that parachutes prevent harm to people jumping out of airplanes.