If you’ve ever read an article about project management, software lifecycle management, or any “new methodology” for developing software, you’ve undoubtedly come across the infamous Standish Group CHAOS Report. (The original 1994 version is available in HTML or PDF directly from the Standish Group).
A column in the August Communications of the ACM (“The Standish Report: Does it Really Describe a Software Crisis?” – ACM membership or subscription required) got me wondering: what did this often-cited report really demonstrate, and on the basis of what data?
The column, Robert Glass’ “Practical Programmer,” points out that “Most such academic papers and guru reports cite the same source for their crisis concern – a study published by the Standish Group more than a decade ago” and argues that “most research studies conducted by academic and industry researchers arrive at data largely inconsistent with the Standish findings.”
Glass points to another article which discovered, in the original Standish Group report, a mention of the researchers methodology, which said they “called and mailed a number of confidential surveys to a random sample of top IT executives, asking them to share failure stories.”
Glass suggests, rightly, that if you start by asking people to share failure stories, you’ll get failure stories. (I could not, however, find this same quotation in the version of the report which is online at The Standish Group, linked to above. Was it later removed?).
Even in the absence of any clear provenance for the data, the report itself is full of vague pronouncements and confused logic.
For example, it breaks projects out into three types:
- 16.2% of projects were completed on time, on budget, and on scope;
- 52.7% were completed, but with “fewer features and functions than originally specified” and were over budget and delivered late
- 31.1% were cancelled at some point during execution
But the description of the second type of projects is problematic: “The project is completed and operational but over-budget, over the time estimate, and offers fewer features and functions than originally specified. ”
Is this a series of “AND” conjunctions which really should be “OR”?
What happened to projects that missed only one corner of the golden triangle (scope, budget and time), or two? What if the project was on time and on budget, but the scope had to change?
The report also claims that “for every 100 projects there are 94 restarts,” adding the qualification that “This does not mean that 94 of 100 will have one restart, some projects have several restarts” – but never actually defining what a restart might be or how they relate to the success or failure of projects.
None of this means, of course, that we should consider software development projects perfect. We’ve all seen projects which go over budget, miss scheduled milestones, or recognize a need to cut scope in order to stay on track. Especially under pure waterfall (all requirements complete and closed before design begins, and so on) methodologies and in larger projects, the need to revise scope later in the project, or revise schedule or cost estimates as the whole picture becomes clearer, is in essence unavoidable.
It certainly suggests, though, that a high degree of suspicion is necessary whenever high level stats about what percentage of IT projects fail.
