Perhaps I have an overly developed sense of smell. Or maybe it's my background in statistics that makes me a skeptic. Or having to sit through hours of vendor presentations about what features will be released in the NEXT version of their product. But when I read Michael Krigman's latest post on Roger Sessions' calculations of how much IT failures cost on an annual basis, it tripped several "are you kidding me?" alarms. So after a brief exchange with Michael on Twitter (@mkrigsman), he suggested that I look at the calculations and see where I disagree with Roger's results.

Fair enough. I spent a few minutes analyzing Roger's argument with simple math and readily available research. Here's the treatment:

According to the World Technology and Services Alliance, countries spend, on average, 6.4% of the Gross Domestic Product (GDP) on Information Communications Technology, with 43% of this spent on hardware, software, and services. This means that, on average, 6.4 X .43 = 2.75 % of GDP is spent on hardware, software, and services. I will lump hardware, software, and services together under the banner of IT.

So far, so good.

According to the 2009 U.S. Budget, 66% of all Federal IT dollars are invested in projects that are “at risk”. I assume this number is representative of the rest of the world.

66% is a fair number. As I blogged earlier this year, project success rate has stagnated at 31% +/-3% over the past 8 years according to Standish Group. The assumption in the second sentence, however, starts leading Roger's analysis off the rails.

A large number of these will eventually fail. I assume the failure rate of an “at risk” project is between 50% and 80%. For this analysis, I’ll take the average: 65%.

That's an interesting assumption that leads to ~43% failure rate for all projects. That's questionable at best, since the actual failure rate of projects was 24% for 2008 (Standish, 2009), and has been fairly constant over the last decade (31% +/- 3% success, 45% +/- 3% challenged, 22% +/- 3% fail). That eliminates roughly 45% of the $6.2T figure, leaving us with $3.46T. However, now that Roger's analysis is fully off the rails, it goes off the beaten path as well:

Every project failure incurs both direct costs (the cost of the IT investment itself) and indirect costs (the lost “opportunity” costs). I assume that the ratio of indirect to direct costs is between 5:1 and 10:1. For this analysis, I’ll take the average: 7.5:1.

Not sure what this assumption is based on. Most organizations will gladly take 30% ROI when embarking on a project. In all of the CBAs that I've had the opportunity to be a part of, intangible benefits were quantified from 30-60% of the overall benefits. So let's take a generous outlier: 60% intangible benefits, 50% ROI. That still works out to... 9:10 opportunity cost to program cost (invest $100, get $150 back, 60% of $150 = $90). To get to 7.5:1, both the ROI and intangible benefit numbers have to be of magnitude rarely seen in business (working backwards: for every $100 spent, get $750 in intangible benefits plus some tangible benefits - where do I sign up?!)

Based on the sober analysis above, the expected cost of IT failures is:

.9 * .24 * .0275 = .00594

of an "average" country's GDP.

So even ignoring the fact that not all countries can be treated the same for purposes of normalization (that would tremendously impact what "average" and "mean" numbers really are) the total cost of IT failures comes out to $414.6B per year (if we use the numbers for GDP in Roger's analysis). That may sound low in comparison to $6.2T, but I would be happy to solve even 24% of that number. What it underscores is that IT project failures are a serious problem that deserves serious treatment.

Aleks

edit: based on @PeterKretzman suggestion, added links to Standish Group's 2009 Chaos Report