Standish Group CHAOS report is one that I follow very closely. From 1994, when the first CHAOS Report came out, Standish has been following success/challenges of projects. Some of the numbers are very familiar to any of us in the software industry - 84% rate of failure/challenges, 45% of features delivered never utilized, and so on. Here's a link to a great interview InfoQ did with Jim Johnson of Standish Group in 2006 with numbers through 2004. Some of the graphs are quite interesting too - for those of you who see patterns everywhere, yes, they do follow the Gartner Hype Cycle curve too closely to be a coincidence. That is a discussion for another day.

So some history - the success rates have improved from 1994 to 2002. That coincided nicely with a tremendous amount of emphasis in project management in a modern enterprise. Improvement is nice, but bad is bad - at the best measurement point in 2002, only 34% of projects actually succeeded. And then, success metrics started dropping again. By CHAOS 2004, 71% were challenged or failed. By 2005, failure rates quoted by PMI were at 72%, and didn't account for projects that were not "too challenged" - close enough is apparently not just for horse shoes in project management. The last two measurements from Standish confirmed that even if the trend wasn't reversed, success metrics have settled near 31% (+/- 3% depending on year). In 2009, it was 32%. So what gives?

I would argue that there are two separate challenges at work here: that IT has been focusing on solving the wrong problem and that measures of success as defined by PMI are insufficient.

First, as an industry, we've invested a lot of money into project and portfolio management. Organizations have spent on methodology development and rollouts, project managers, on project management software, on portfolio management software, on expense management software, on vendor/procurement management, on... well, if it was in the PMBOK, it was justified. And if there is anything to be gleaned from the CHAOS Report numbers, is that all this spending was actually solving the 20% problem, and maybe not even 20%. For all the investment, the project success rates improved from 16% to 31% over the first 8 years. And they've been there for the last 8 years. It seems project management has reached the plateau of productivity, to use the Hype Cycle metaphor, and further investment into the space is coming up against the law of diminishing returns. That is of little consolation to a CIO who goes into annual planning armed with knowledge that two out every three projects that will be approved will not deliver on their clients' expectations.

Second, there may be another explanation. Somehow, some way, modern enterprises keep on working. Sure, some of that is due to spiderwebs of manual processes created to pick up slack for failing projects and missed requirements. But things do work, if inelegantly so. Based on that, some have questioned Standish Group's method for classifying projects as successful, failing or challenged. What if the aim is off - perhaps the real issue is in the success criteria of the projects themselves? Consider these two points:

1. Research suggests that 84% of organizations either do not do business cases for IT projects or do them for a select few key projects. (Gartner) Of the 16% that are disciplined in their technology investment decision practices, very few have organizational mechanisms in place to measure and manage to the business case once approved.
2. As I look on the bleak landscape painted by the analysts, we have come across case studies where an initiative should be judged a failure or challenged by conventional project management metrics. Yet they deliver tangible value in areas that were not identified, or could not have been identified when the initiative was approved.

The first point highlights that very few organizations definitively know the actual disposition - be it success, failure, or something in between - of all their projects. The second point highlights that while the concept of "project" is fairly well-defined, the success criteria are not, and that a more appropriate way to judge whether a project succeeded is to look at the short, medium, and long-term impact it has on existing and new assets. Of course, since 89% of organizations have virtually no metrics in place to measure IT effectiveness except for finance’s account of expenses (Gartner), measuring that impact can be even more challenging than successfully delivering a project.

Aleks