New ‘smart’ machines could fundamentally change how people interact with computers
A new type of "smart" machine that could fundamentally change how people interact with computers is on the not-too-distant horizon at Sandia.
Over the past five years a team led by Labs cognitive psychologist Chris Forsythe (15311) has been on the fast track in developing cognitive machines that accurately infer user intent, remember experiences with users, and allow users to call upon simulated experts to aid them in analyzing situations and making decisions.
Although work is being coordinated through the Emerging Threats Strategic Business Unit (SBU), the team includes representatives from organizations 5838, 8964, 9212, 9216, 12335, 15221, 15222 and 15311, as well as university and industry collaborators.
"In the long term, the benefits from this effort are expected to include augmenting human effectiveness and embedding these cognitive models into systems like robots and vehicles for better human-hardware interactions," says John Wagner, Manager of Computational Initiatives Dept. 15311. "We expect to be able to model, simulate, and analyze humans and societies of humans for DOE, military, and national security applications."
The program started with an effort led by Sabina Jordan (5838) on the Next Generation Security Simulation project. Chris developed a framework for constructing individualized computer models that simulated how people apply their knowledge to make decisions in real-world settings. Subsequently, these developments provided the basis for an internally funded Laboratory Directed Research and Development (LDRD) grant through the Advanced Concepts Group in which the computer model was elaborated to include the influence of organic factors such as arousal and emotion.
Synthetic human
The initial goal of the work was to create a "synthetic human" — software program/computer — that could think like a person.
"We had the massive computers that could compute the large amounts of data, but software that could realistically model how people think and make decisions was missing," Chris says.
There were two significant problems with modeling software. First, the software did not relate to how people actually make decisions. It followed logical processes, something people don’t necessarily do. People make decisions based, in part, on experiences and associative knowledge. In addition, software models of human cognition did not take into account organic factors such as emotions, stress, and fatigue — vital to realistically simulating human thought processes.
In the first LDRD project, Chris developed the framework for a computer program that had both cognition and organic factors, all in the effort to create a "synthetic human."
In 2001 two other LDRD grants were awarded. One, part of the Nonproliferation and Materials Control SBU, was to develop methodologies that allowed the knowledge of a specific expert to be captured in the computer models. Through this project, cognitive psychologist Ann Speed (12335) has developed unique approaches for obtaining both explicit and implicit knowledge and translating it into quantitative data necessary for constructing a computer model (see "Cognitive Collective" on page 5).
The second was for the Emerging Threats SBU to include episodic memory — memory of experiences — in the software. This would allow a synthetic entity to apply its knowledge of specific experiences to solving problems in a manner that closely parallels what people do on a regular basis.
Strange twist
Chris says a strange twist occurred along the way.
"When I got the second LDRD grant, I needed help with the software," Chris says. "I turned to some folks in Robotics [Patrick Xavier (15221) and David Schoenwald (9216)], bringing to their attention that we were developing computer models of human cognition."
The robotics researchers immediately saw that the model could be used for intelligent machines, and the whole program emphasis changed. Suddenly the team was working on cognitive machines, not just synthetic humans.
Work on cognitive machines took off in 2002 with a contract from the Defense Advanced Research Projects Agency (DARPA) to develop a real-time machine that can infer an operator’s cognitive processes (see "Airborne Warning and Control simulation" below).
"This project is developing technology to fundamentally change the nature of human-machine interactions," Chris says. "Our approach was to embed within the machine a highly realistic computer model of the cognitive processes that underlie human situation awareness and naturalistic decision making. Systems using this technology are tailored to a specific user, including the user’s unique knowledge and understanding of the task."
The idea borrows from a very successful analogue. When people interact with one another, they modify what they say and don’t say with regard to what the person knows or doesn’t know, shared experiences, known sensitivities, etc. The goal is to give machines highly realistic models of the same cognitive processes so that human-machine interactions may enjoy benefits similar to human-human interactions.
Recently a major car company has taken interest in real-time cognitive machines. The technology could adjust systems such as the brakes in response to the driver’s cognitive state — talking on cell phone, or changing the radio — in real time. Also, a developer of PC desktop software applications has expressed an interest in the capability for the computer to know what a user has done in the past so that current activities can be put in the context of experience. Computers would have a record of all their interactions with a user so that if a user starts to change a setting, it could tell him that he tried this before and it didn’t work.
"It’s entirely possible that these cognitive machines could be incorporated into most computer systems produced within 10 years," Chris says.
Grand Challenge
Early this year work began on a Next Generation Intelligent System Challenge LDRD project. Russ Skocypec (15310) is the program manager, and Larry Ellis (6502) is the principal investigator.
"The goal of this Grand Challenge is to significantly improve the human capability to understand and solve national security problems, given exponential growth of information and very complex environments," says Larry. "We are integrating extraordinary perceptive techniques being developed by John Ganter (6533) and his team with Chris’ cognitive systems."
The intent of the cognition track of this project is to develop technology that will augment the capacity of analysts, engineers, war fighters, critical decision makers, scientists, and others in critical jobs to detect and correctly interpret meaningful patterns based on large volumes of real-time and archival data derived from diverse sources.
This may involve real-time systems that use simulated experts from different domains that singly and collectively assess immense volumes of data to alert engineers and analysts to potential problems.
The same technology would allow an individual to visualize the knowledge of experts and even compare experts to one another.
"We have already shown that engineers from different disciplines have quite different knowledge structures that allow them to look at the same data and reach somewhat different conclusions," says Chris. "Our intent is to capture these unique perspectives in a practical set of tools that may in essence allow any given engineer to function as a team of engineers with different expertise and experience."
A primary emphasis of the Grand Challenge involves development of tools that will enable individualized knowledge to be captured by software running in the background, without the need to directly interact with the engineer or analyst. Currently, the rigorous methods required to develop an accurate model of an individual’s knowledge of a domain are very labor-intensive. "Automated knowledge modeling," says Chris, "is the most challenging aspect in developing this technology."
However, the ability for a machine to automatically acquire accurate models of users’ knowledge could have dramatic impacts. In the near-term, this would enable adaptive help systems that adjust to the specific knowledge of a user or technologies that allow a novice user to compare their knowledge to that of one or more experts.
"Looking into the future," Chris adds, "one may envision an economy in which an individual’s knowledge and experience may be packaged and sold as a commodity — when an engineer buys their computer-aided design software they may select from a library of experts whose cognitive models come installed with the software not unlike going to a music store and selecting a stack of CDs of your 10 or 12 favorite artists."
Russ sees an exciting future for this work and the impact it can have on Sandia
"One of Sandia’s strengths is our expertise in understanding and representing complex physical behavior from a sound foundation basis," Russ says. "The efforts that we are now undertaking in cognition are beginning to lay a similar foundation upon which we will build capabilities to represent human behavior, which is a difficult, yet critical, aspect of national security."