You can’t feel it, see it, or touch it, but by leaps and bounds it is becoming a dominant force in our lives. "It" is software that is found in everything from personal computers to complex military systems. Because of this growing reliance on software and demands that it be fail-proof — particularly in critical areas such as weapons — Sandia has undertaken a new initiative to improve the way software is developed at the Labs.

Called the Software Improvement Initiative, it uses the Software Engineering Institute’s (SEI) Capability Maturity Model (CMM) to give software-intensive organizations a path to improve — changing from an ad hoc, immature process of developing software to a mature, disciplined one.

"As we rely more and more on software, it becomes increasingly important for it to be developed following logical processes that can be retraced," says Ron Detry, Director of Nuclear Weapons Programs Integration and Studies Center 9800 and long-time quality processes champion. "CMM gives you some assurance that your software will work because you are following a logical way of development that has proven successful in the past."

A CMM appraisal results in five possible levels of software development maturity. The lowest is level 1, where the process is unpredictable and poorly controlled. The highest is level 5, where the process has been measured and controlled and is in a continuing improvement stage. Specific key processes must be followed to achieve each level.

Several Sandia centers have started on their CMM journey, Ron says. One reason is that it is seen as a way to improve software development processes and ultimately the final product. Just as strong a driver, however, is that achieving at least a level 3 may soon be mandated by the Department of Defense (DoD) and Lockheed Martin.

Already part of DoD is requiring that software-significant organizations — contractors whose primary function is to provide software — be at level 3. Since Sandia is not a commercial supplier of software for DoD, the Labs doesn’t have to meet the requirement at this time.

Mandate on horizon

Many at the Labs, however, see the mandate on the horizon — especially because SEI, which developed the levels and standards, is a federally funded research and development center sponsored by DoD. Also, Lockheed Martin as a whole is challenging its "software significant" organizations to a level 3 or higher.

Because it consisted of software quality engineers and was already involved in quality assurance, Quality Engineering Dept. 12326, headed by Mike Blackledge, was tapped to help create a CMM-based assessment system and then go to organizations and assess their software development methods. Working with Mike and his department members were representatives of SEI and cooperative research and development agreement (CRADA) partner SEMATECH, a consortium of major semiconductor manufacturers.

Feet wet with SEMATECH

Dave Peercy, Dwayne Knirk, Patty Trellue, and other Dept. 12326 members got their feet wet doing assessments of software development companies associated with SEMATECH. The Sandians worked with companies in Silicon Valley and the Boston area. In 1998 and 1999 they did an assessment of DOE’s Accelerated Strategic Computing Initiative (ASCI) code teams at Sandia, Los Alamos, and Lawrence Livermore national laboratories.

Interest in CMM didn’t fully catch on at Sandia until June 1999, when Mike invited the chair and vice chair of the Lockheed Martin Software Council to the Labs to present a forum on the company’s success with the software development framework. They discussed how it improved productivity and cost effectiveness. Their words captured the attention of a couple of vice presidents and several directors at Sandia whose organizations are involved in software development. Soon Mike’s department was called to do assessments of centers throughout the Labs. In addition to the ASCI centers, they’ve visited Information Systems Engineering Center 6500, Information Systems Development Center 9500, and High Integrity Software System Dept. 2662. Sandia has opted to perform self-assessments, using Mike’s group as the assessors. If DoD, DOE, or Lockheed Martin mandate CMM, the Labs would have to bring in official assessors certified by SEI.

Requirements

The big difference, says Jim Rice, Director of Information Systems Engineering Center 6500, is that CMM operates in a pass/fail mode — if an organization doesn’t meet every requirement of a level, it remains at the lower level. Using Mike’s grading method, which assigns numerical ratings in each area being assessed, the organizations know exactly where their strengths and weaknesses are. Interestingly, the Information Systems Engineering Center 6500 and the Information Systems Development Center 9500 fell short in exactly the same two process areas — software quality assurance, a level 2 requirement, and peer review, a level 3 requirement.

Paul Merillat, Director of Center 9500, has a theory about why this is so.

"It deals with the level of trust Sandians have toward each other," he says. "Sandia culture says ‘I trust you to do good work.’ Thus, they don’t see a need for peer reviews — reviewing someone else’s work — or quality assurance reviews, verifying that the required work products were actually generated."

Both Paul and Jim say that following the CMM system makes life easier for people involved in software development.

"In the past software development has been hero-based," Paul says. "People would come and devote 11 hours a day to solve a problem and get it right. Then they get burned out. Not only is this disruptive, but it’s costly." Each CMM level requires documented processes that if followed afford at least the opportunity for software engineers to get the job done on time without working day and night, reducing stress.

"The purpose is to improve the product — to get better results," Jim says. "By having effective processes coupled with smart, skilled people, you can achieve that. Good people with good processes can accomplish amazing things."

Last modified: August 18, 2000