Sandia helps bridge the gap between technology and human readiness

Three words piqued systems engineer Judi See’s interest during a presentation at Sandia late one afternoon: human readiness levels.

“My ears immediately perked up. The speaker didn’t really say anything more about it,” Judi said, referring to a 2015 presentation by the National Defense Industrial Association Human Systems Division chair. “I was intrigued, so I went online and started looking up human readiness levels. The concept was first proposed back in 2010.”

That curiosity eventually helped fuel the development of a national standard designed to answer a question that engineers do not always measure as early or as consistently as technical performance: Is a system ready for the people who will use it?

The American National Standards Institute and Human Factors and Ergonomics Society published the standard in 2021. The standard must be reaffirmed or reviewed every five years, and Judi has been collecting feedback for improvement from current users for potential updates later this year.

Big benefits

Prior to Judi’s curiosity, the military worked on human readiness levels but not to the point of implementation.

“The human readiness level scale evaluates whether any technology under development is compatible with human limitations and capabilities,” said Judi, who works in strategic analysis for nuclear deterrence. “It looks at all aspects of effective and safe human use.”

The human readiness level scale, or HRL, is the counterpart to the technology readiness level scale, or TRL, developed by NASA in the 1970s to communicate technical maturity.

The HRL puts humans front and center during development of technology, instead of waiting until it’s fully developed to get user feedback.

“When the design is set in stone, what you’re left to do is train the user,” Judi said. “Making changes later in the development cycle is costly.”

Studies show big benefits when human readiness is incorporated at the beginning of technological development. It improves the overall effectiveness of that technology, improves timelines, reduces program risks and perhaps most importantly, reduces risk for human error and accidents.

Sandia study

“I proposed conducting a study on the utility of the HRL scale here at Sandia within our nuclear weapons programs,” Judi said. “It got approved.”

Judi completed the study and the response was clear: people involved in the weapons programs saw value in a tool that would keep the human element from becoming an afterthought.

“They thought it would have utility and it would help them think about the human in the system more consistently throughout the entire program,” she said.

Shortly after she completed the study, Judi got a call from a systems engineering professor at Old Dominion University who had seen her work on the HRL scale.

“As we talked, we realized this was still conceptual,” she said. “If we wanted to get something implementable like the TRL scale, we needed additional work.”

Leading experts

Outside of her regular Sandia duties, Judi formed a joint working group in 2019. Others at Sandia also directly contributed to the effort.

“Maturing the human readiness scale and chairing the writing committee for the standard required a tremendous amount of work,” Judi said. “I believed in its importance. Passion and vision can be great drivers. They enabled me to get other people on board, many of whom I had never even met.”

More than 30 human systems experts from across the country, including those with the DOE, national laboratories, the military, industry and academia, worked to develop the HRL scale.

Mica Endsley, a former chief scientist of the U.S. Air Force, participated in the working group. “It’s been a tremendous effort,” she said. “It lets people understand intuitively how much work has been done.”

Endsley added that when working with generals at the Pentagon and other decision-makers, they understood the TRL scale, which was implemented there in the late 1990s. “We tried to map to those levels,” she said.

The TRL focuses on technical maturity and includes nine levels. Level 1 is a basic concept or idea. Level 9 means it’s a fieldable technology or system. The HRL scale is intended to align directly with those TRL levels. For example, at Level 4, while the TRL scale addresses component or breadboard validation in the lab, the HRL scale emphasizes comparable human-centered lab work consisting of modeling, part-task testing and trade studies to explore candidate human systems design concepts for the technology under development.

Creating consistency

While users can subjectively say whether a technology is usable, the detailed criteria and evidence in the HRL standard enable human factors engineers to represent human readiness with a single numerical measure that anyone — across industries and across the world — will understand. In addition to standardization, the importance of documenting evidence before moving to the next readiness level on the scale is emphasized. In an ideal world, Judi said, technical maturity and human readiness advance together.

“While you’re making your technical improvements, you’re also at the same time thinking about the user of that technology,” she said.

After about a year, the working group fully defined all nine levels of the HRL scale, established entry and exit criteria for each level, and completed tabletop exercises to evaluate its utility for historical scenarios involving human systems issues.

Formalizing the scale

After developing and defining the nine levels of the HRL scale, the team turned its focus to formalizing the framework.

“We then decided to form a committee to turn it into a formal technical standard through the American National Standards Institute and Human Factors and Ergonomics Society,” Judi said. “To take it from essentially a tested prototype to that final fieldable product took organization, dedication and persistence in the face of setbacks.”

Judi said one lingering question after publication of the standard was whether any organization would adopt or implement it. That question was soon answered when the FY25 National Defense Authorization Act required the DOD to review the HRL scale as a way to enhance safety in relation to human factors.

“Seeing the military recognize the value of using the standard and officially adopting it felt like a great accomplishment,” Judi said. She added that this requirement could be expanded to contractors and partners.

Pam Savage-Knepshield, a senior human factors engineer at defense contractor CACI International Inc., has applied the HRL scale to four software programs of record for Army field artillery command and control systems. She said that when used alongside the TRL scale, the HRLs help validate human performance before a system reaches the battlefield, strengthening both program success and mission readiness.

“The HRL scale is especially critical to the application of military equipment design because the stakes are high on the battlefield,” Savage-Knepshield said. “Warfighters’ lives and mission success demand intuitive, safe and effective equipment that appropriately accommodates and augments human capabilities and limitations. HRLs act as a security net to ensure equipment is truly battlefield-ready and optimally supportive of the personnel using it.”

Promoting the scale

Endsley, the chair of the Human Factors and Ergonomics Society Government Relations Committee, has been working to raise awareness at other government agencies about the standard, and most have been receptive.

“We’re promoting that the standard is best practice,” Endsley said. She emphasized that the scale can be used on any technology that is under development or being considered for procurement. “Human factors need to be considered early to save money and time.”

As a result of this awareness work, Congress also required the Federal Aviation Administration to review the HRL scale and standard. Judi said the FAA is currently finalizing how it will incorporate the standard into its work after completing extensive reviews.

Meanwhile, Judi is frequently invited to chair conference panels, tailor presentations and advise agencies to consider how HRL implementations can help meet their organizational goals of mission-ready, safe systems.

“I typically receive requests from organizations throughout the U.S. every year to present the human readiness level scale and the standard to their human systems groups,” she said.

The Human Factors and Ergonomics Society recognized Judi in 2022 with the Oliver Keith Hansen Outreach Award for her work on the HRL scale.