What's in a CMM rating?
Does hiring a CMM Level 5 service provider guarantee that an outsourced software project will come in on time and on budget?
Will a higher CMM rating automatically mean higher costs?
What impact does earning a rating have on software quality?
These are just a few of the questions confronting IT managers charged with contracting out an increasing volume of application development and maintenance work to lower-cost offshore outsourcers.
Developed by the Software Engineering Institute (SEI) at Carnegie Mellon University in Pittsburgh, CMM -- short for Capability Maturity Model -- is a set of rigorous standards for software development that's based on five levels. Of some 70 companies worldwide that have publicly acknowledged reaching the highest rating of Level 5, about 50 are in India, according to the SEI and Gartner.
Not surprisingly, these Indian outsourcers aggressively tout their CMM rating, marketing themselves as top-notch developers with standardized, repeatable processes in place for delivering the highest quality software. Executing standardized processes also works to keep down costs, enabling Level 5 providers to pass on additional savings to customers, according to Sangita Singh, head of strategic marketing at Wipro, an Indian outsourcing company with U.S. headquarters.
Research confirms that higher CMM levels correlate with fewer software defects. But the highest CMM rating doesn't necessarily guarantee the greatest savings for customers. "The data on quality and maturity levels shows there is a definite improvement in costs and (on-time project completion) schedules," says Bill Peterson, program director for software engineering process management at the SEI. "But whether the supplier passes the savings on to the buyer, we don't know. That's more business than anything to do with the logic of costs.
"What we are saying is that as a Level 5, (suppliers) are better and they're able to charge more, not less," Peterson adds.
At the same time, a Level 5 CMM rating comes with no guarantees, and in some cases, it may even be overkill, experts say.
"CMM is a great discipline, and it is a great designation to have," says Bart Perkins, a Computerworld columnist and managing partner at Leverage Partners, which helps CIOs manage IT suppliers. "But the reality is that if an outsourcer is at Level 5 and the client is at Level 1 or 2, the client doesn't have the internal discipline to take advantage of the Level 5 provider's standardized routines."
Defining system or project requirements is a prime example. "With CMM, the entire requirements process is very rigidly defined. A Level 5 requirements document is very detailed and explicit and has metrics associated with it," Perkins explains. "But a company at a CMM Level 0 or 1 could have their requirements on the back of an envelope and no metrics. The Level 1 companies are lucky if they write out two pages."
The upshot, says Perkins, is that touting a CMM Level 5 rating to a Level 1 buyer "comes down to touting a feature that's of little value. It's like a car salesman in Alaska touting a car's great air conditioning. It may be great, but you can't take advantage of it."
Yet some companies, such as Farmers Insurance Group in Los Angeles, contract with Level 5 outsourcers exclusively, even though they may be unable to reap all of the benefits of doing so.
"The CIO dictated that we only do business with CMM Level 5 partners. It was a way of distinguishing the best companies from the rest of the pack," explains Alan Stanley, a program manager at Farmers.
"Beyond that, we don't take advantage of CMM. We tend to dictate how we want work done. We allocate work and processes based on what we do here, so I don't think we've really benefited from the CMM Level 5 side," he adds.
Helen Cousins, former CIO at Cendant, says she believes that hiring a Level 5 outsourcer is a way to raise the bar for your own IT organization. "One of the things we gained out of necessity is the ability to more clearly define what we want," says Cousins, who is now CIO at Dex Media. "I've also noticed that when people working side by side are with people who are disciplined, it starts rubbing off."
But in a January 2003 report on the subject, Gartner analyst Partha Iyengar cautioned that users should also remember that CMM standards are descriptive rather than prescriptive, meaning that "they describe what must be done, rather than how it must be done." Consequently, a vendor can specify a certain way of executing a process that isn't the best possible implementation of that particular process.
In other words, Iyengar says, "CMM standards certification in no way guarantees that a vendor's internal implementation of these standards is best-in-class in any way."