- By Thomas R. Temin
- Aug 21, 2002
Thomas R. Temin, GCN editorial director
Can it be true? Is even the vaunted Capability Maturity Model for software development so subjective that you can't always believe vendor claims of CMM level ratings?
As Susan M. Menke reported in the Aug. 12 issue, it's a lot easier to claim a given maturity level under the Software Engineering Institute system than to verify it. That's because the levels are conferred by SEI-certified auditors, not the institute itself.
In the 1960s, transistors and mass production hit the sound equipment industry and, like PCs later on, suddenly everyone was buying stereo hi-fi gear. Manufacturers would make fantastically inflated claims about the wattage of their amplifiers. Under some rating systems, they could advertise that a 25-watt amp had 150 watts of 'peak music power.' Those who didn't know their sine waves were doomed to disappointment when they turned the things on.
Eventually, as knowledge diffused about the realities behind such claims, amplifier power, like specs in so many other industries, settled into a more comparable reporting process.
But software development rating can't be so easily quantified. The lesson is, buyer beware, even with a reasonably sound rating method such as CMM.
For government agencies buying software development and wanting some semblance of a predictable and favorable outcome, the CMM rating is a good place to start. Lloyd Mosemann, former software chief of the Air Force and a proponent of software quality, is calling for agencies to up their software acquisition maturity so they're on equal footing with vendors touting this or that CMM rating.
Until your agency gains that knowledge, there are certain questions you should always ask a would-be vendor:
- When did you get your rating, and what do you do to maintain it?
- Are the developers responsible for your CMM ratings the ones who will work on my project? Do they still work there?
- Are you willing to submit a risk mitigation plan with this proposal if you are awarded a contract?
Whatever a vendor's CMM rating, agencies aren't absolved from careful oversight.