RSS Feed
Thursday, October 22, 2009 | 
A spirited conversation on Twitter with @PeterKretzman and Roger Sessions (@RSessions) led to a couple of updates:
Factor = 5.4 * .24 * .0275 = .03564
Total Yearly Loss: .03564 * $69.8T = $2.487T (round up $2.5T)
Editorial Comment: While $6.2T seemed too high , and $414B seemed a bit low, could this be the real number? Details below:
1. Updates to my math:
Roger's definition of indirect costs adds expended cost of failed project to both tangible and intangible benefits. So, to use my example of 50% ROI, 60% of benefits are intangible example, the multiplier would be 100% + 150% = 250% or 2.5.
That thought experiment assumes a very strong business case. Available research further confirms that experience: "Nucleus case studies reveal that, on average, indirect benefits account for half of technology ROI. " (Nucleus Research, 2004). In my experience, most organizations have hurdle rates around 20%, not 50%, so very few approved projects will have benefit case with 50% ROI. So, let's take 20% as our floor (2.2 multiplier). Example cited in Roger's white paper seems highly unusual, but it does provide a ceiling (9.6 multiplier). A question without a good answer is what is the distribution of project spend between 2.2 and 9.6 - my hunch is that even if it is a bell curve (normal distribution), it's far more weighted toward the floor in terms of number of projects, while far more weighted toward the ceiling in terms of spend. If we assume those two cancel each other out, the average multiplier becomes 5.4. I'd be interested if anyone has done a study to support/update this number, rather than using an assumption.
2. Updates to Roger's math:
I still find it curious that Roger and I are using the exact same research note from Standish Group to get the ratio of failed projects - but using different metrics from that note. I take the definitions from Standish more literally - it's hard for me to argue that their definition of "failed" is not sufficient, especially since they clearly delineate between "failed" and "challenged". So I'll stick with 24% failed metric from 2009 report. Whether there are some incurred indirect costs of "challenged" projects is an open question - it makes sense that there would be, but I'd love to see a study rather than make that assumption.
3. Open questions
First, it would be nice to have a metric supported by research on what the multiplier should be. That is the biggest weight in the equation, and it needs to be more than assumptions on all sides.
Second, there are obviously indirect costs to challenged projects. I'd be interested to see whether anyone (ahem, PMI?) has done a study that documents the costs of missing deadlines. That will also have an impact on the final answer, though probably not as much.
5 comments:
It's nice to see our numbers starting to come closer together. Let me suggest two factors that might account for our remaining differences.
First, Aleks is focusing only on the costs to the organization. I have been considering the cost to the entire economy. Economy costs include costs to others outside of the company.
Consider putting in an SOA for a supply chain. A failure here would drive direct costs of the cost of the SOA. The indirect costs (as used by Aleks) includes the cost of lost revenue had the SOA been successful. Indirect costs (as used by me) includes interest on the cost of the SOA, opportunities that could have been driven by the increased revenue had the SOA been successful, the cost to our customers of using a less optimized approach to ordering from us, and the cost of lost customers who eventually gave up on our system.
By the way, the Nucleus Research report that Aleks quotes does not define indirect costs, so I have no idea what they mean or how they acquired their data.
Second, with respect to the Standish Report. I discuss this more in my white paper, but the Standish report is based on failure rates of IT projects. My numbers are based on failure rates of IT budgets. Standish numbers are really not helpful in understanding IT effectiveness.
Let me give an example of the problems with Standish. Suppose you are an IT department with a $1M budget. Say you have seven IT projects completed this year:
5 @ $50K
1 @ $100K
1 @ $700K
Of all these projects, which is the most likely to fail? All other things equal, the $700K project, because it is the largest and most complex.
Let's assume the following:
- of the $50K projects, 4 succeed and 1 fails.
- the $100K project succeeds.
- the $700K project fails.
Standish would report this as 5/7 success rate, or a 71% success rate, or a 29% failure rate.
I would report this as $300K of $1M budget invested in successful projects, $700K invested in failed projects for a success rate of 30% and a failure rate of 70%.
So both Standish and I are looking at the same numbers, yet we have opposite conclusions. Whose interpretation is better?
I argue that, from the organizational perspective, my interpretation is much more reasonable. The CEO wants to know how much money is being spent and how much is being made, not how well the IT department does one-off projects, which are exactly the projects that dominate the Standish numbers.
So the bottom line is that I still stand by my numbers as reported in my White Paper, IT Complexity Crisis http://bit.ly/3O3GMp.
But I also really appreciate Aleks looking closely at the numbers and challenging me. I mean this seriously. It is only through this process of challenge and response that we can all have confidence that we are approaching a shared understanding. I know I have learned a lot in responding to Aleks's challenges and I hope you have too in reading our exchange.
- Roger
As you have probably noticed, my math in the Standish example, is very slightly off, but not enough to change my conclusions. Sorry about that!
Roger -
A few thoughts -
1. In my experience, non-project portion of IT budget is not really at same risk of "failure" as projects are. Most organizations generally got maintenance bucket under pretty tight control. While maintenance is an outcome of projects, failed or successful or something in between, that is usually captured as part of cost-benefit analysis.
2. Benefits are more than revenue, as my example clearly identifies. Furthermore, per your example of failed SOA effort:
- Interest on cost of SOA is already accounted for in the hurdle rate (by definition)
- From a management perspective, I think most people would look at what you're defining as opportunity cost as sunk cost. Perhaps the challenge here is that CBAs are not systematically done for every project in most organizations? There are several approaches to calculating this component, although I'm not sure as to their utility. Continually telling people that life could have been so much easier if they were only successful on that one giant failure (or series of smaller failures) is unlikely to endear the messenger to their audience. What is the opportunity cost of lost productivity due to deprecated morale?
- Customer attrition is usually captured by the "do nothing" option that is standard in our TCO models. That is obviously used w/n benefits tab of the cost benefit analysis.
- So all that is unaccounted for is 'cost to customers of using suboptimal system'. I would argue that this is also taken into account - by the market. Companies don't live in a vacuum - there are competing organizations who are actively campaigning for those customers' business. So this cost component is very temporary, especially in commodity services and products as clients switch vendors and providers.
Working on thoughts 3 and 4 right now,
Regards,
Aleks
Continued...
3. To my knowledge, nobody has really provided "the" definition for indirect costs - that's why I think this conversation is so important. Nucleus Research number on intangible benefits is just one portion of the overall number, albeit a very important one. Intangible benefits (like brand equity, future reuse opportunities, etc) provide a good constraint for our calculations. And as I mentioned in the post, I would want every component of the formula to be backed up with research, rather than assumptions.
4. I've never seen a project portfolio similar to your thought experiment. An IT department with a $1M budget is lucky to have 60% of that available for new development projects. Let's say that $1M budget is for projects only to align with your example. How many CIOs (and their peer C-level execs) would allow 70% of the total project budget to be tied up in a single project? It is certainly not state of the practice - that single large effort would likely be split up in to a series of smaller initiatives to guard against complete failure.
Organizations I've been a part of usually have a threshold for strategic projects that is less than 50% of the overall project budget - and that's for all strategic projects, not a single one. That's what is behind my reasoning that by Standish Group's numbers are reasonably solid in the post - even if it's not a "true" bell curve. Putting all (or most) of the eggs in one basket is something (I hope) most business and technology leaders know not to do!
Regards,
Aleks
Great discussion, Aleks and Roger. Combine the present lack of economics know-how on the part of most EAs with the inherent slipperiness of such concepts as 'indirect cost' - you have us all rushing back to the few textbooks we may still have around. Being a TOGAF guy, I'm heartened by the increased emphasis on economic factors in v9 vs v8. I believe there is much more we need to add, as your discussion clearly demonstrates.
Regards
John Polgreen
Post a Comment