We present a high-level architecture for how artificial intelligences might advance and accumulate scientific and technological knowledge, inspired by emerging perspectives on how human intelligences advance and accumulate such knowledge. Agents advance knowledge by exercising a technoscientific method—an interacting combination of scientific and engineering methods. The technoscientific method maximizes a quantity we call “useful learning” via more-creative implausible utility (including the “aha!” moments of discovery), as well as via less-creative plausible utility. Society accumulates the knowledge advanced by agents so that other agents can incorporate and build on to make further advances. The proposed architecture is challenging but potentially complete: its execution might in principle enable artificial intelligences to advance and accumulate an equivalent of the full range of human scientific and technological knowledge.
The Tularosa study was designed to understand how defensive deception-including both cyber and psychological-affects cyber attackers. Over 130 red teamers participated in a network penetration task over two days in which we controlled both the presence of and explicit mention of deceptive defensive techniques. To our knowledge, this represents the largest study of its kind ever conducted on a professional red team population. The design was conducted with a battery of questionnaires (e.g., experience, personality, etc.) and cognitive tasks (e.g., fluid intelligence, working memory, etc.), allowing for the characterization of a “typical” red teamer, as well as physiological measures (e.g., galvanic skin response, heart rate, etc.) to be correlated with the cyber events. This paper focuses on the design, implementation, data, population characteristics, and begins to examine preliminary results.
Exposure to extreme environments is both mentally and physically taxing, leading to suboptimal performance and even life-threatening emergencies. Physiological and cognitive monitoring could provide the earliest indicator of performance decline and inform appropriate therapeutic intervention, yet little research has explored the relationship between these markers in strenuous settings. The Rim-to-Rim Wearables at the Canyon for Health (R2RWATCH) study is a research project at Sandia National Laboratories funded by the Defense Threat Reduction Agency to identify which physiological and cognitive phenomena collected by non-invasive wearable devices are the most related to performance in extreme environments. In a pilot study, data were collected from civilians and military warfighters hiking the Rim-to-Rim trail at the Grand Canyon. Each participant wore a set of devices collecting physiological, cognitive, and environmental data such as heart rate, memory, ambient temperature, etc. Promising preliminary results found correlates between physiological markers recorded by the wearable devices and decline in cognitive abilities, although further work is required to refine those measurements. Planned follow-up studies will validate these findings and further explore outstanding questions.
The Rim-to-Rim Wearables At The Canyon for Health (R2R WATCH) study examines metrics recordable on commercial off the shelf (COTS) devices that are most relevant and reliable for the earliest possible indication of a health or performance decline. This is accomplished through collaboration between Sandia National Laboratories (SNL) and The University of New Mexico (UNM) where the two organizations team up to collect physiological, cognitive, and biological markers from volunteer hikers who attempt the Rim-to-Rim (R2R) hike at the Grand Canyon. Three forms of data are collected as hikers travel from rim to rim: physiological data through wearable devices, cognitive data through a cognitive task taken every 3 hours, and blood samples obtained before and after completing the hike. Data is collected from both civilian and warfighter hikers. Once the data is obtained, it is analyzed to understand the effectiveness of each COTS device and the validity of the data collected. We also aim to identify which physiological and cognitive phenomena collected by wearable devices are the most relatable to overall health and task performance in extreme environments, and of these ascertain which markers provide the earliest yet reliable indication of health decline. Finally, we analyze the data for significant differences between civilians’ and warfighters’ markers and the relationship to performance. This is a study funded by the Defense Threat Reduction Agency (DTRA, Project CB10359) and the University of New Mexico (The main portion of the R2R WATCH study is funded by DTRA. UNM is currently funding all activities related to bloodwork. DTRA, Project CB10359; SAND2017-1872 C). This paper describes the experimental design and methodology for the first year of the R2R WATCH project.
In many settings, multi-tasking and interruption are commonplace. Multi-tasking has been a popular subject of recent research, but a multitasking paradigm normally allows the subject some control over the timing of the task switch. In this paper we focus on interruptions—situations in which the subject has no control over the timing of task switches. We consider three types of task: verbal (reading comprehension), visual search, and monitoring/situation awareness. Using interruptions from 30 s to 2 min in duration, we found a significant effect in each case, but with different effect sizes. For the situation awareness task, we experimented with interruptions of varying duration and found a non-linear relation between the duration of the interruption and its after-effect on performance, which may correspond to a task-dependent interruption threshold, which is lower for more dynamic tasks.
Research was undertaken to gain an understanding of the interplay between cyber security professionals and the software tools utilized in performing their jobs. Substantial investments are devoted to purchasing and developing software tools targeting cyber security operations. However, development is largely based on anecdotal knowledge concerning the work processes, cognitive demands, and the needs and requirements of cyber security analysts. The current study first characterized the workflow of a Cyber Security Incidence Response (CSIRT) team, including their use of software tools, and instantiated this workflow within a simulation model. Next, data was collected during cyber security training exercises reflecting the use of software tools. It was discovered that while cyber security professionals rely heavily on specialized software tools, their jobs require that they effectively integrate the use of specialized software tools with the use of general- purpose software tools.
Adaptive Thinking has been defined here as the capacity to recognize when a course of action that may have previously been effective is no longer effective and there is need to adjust strategy. Research was undertaken with human test subjects to identify the factors that contribute to adaptive thinking. It was discovered that those most effective in settings that call for adaptive thinking tend to possess a superior capacity to quickly and effectively generate possible courses of action, as measured using the Category Generation test. Software developed for this research has been applied to develop capabilities enabling analysts to identify crucial factors that are predictive of outcomes in fore-on-force simulation exercises.
Within large organizations, the defense of cyber assets generally involves the use of various mechanisms, such as intrusion detection systems, to alert cyber security personnel to suspicious network activity. Resulting alerts are reviewed by the organization's cyber security personnel to investigate and assess the threat and initiate appropriate actions to defend the organization's network assets. While automated software routines are essential to cope with the massive volumes of data transmitted across data networks, the ultimate success of an organization's efforts to resist adversarial attacks upon their cyber assets relies on the effectiveness of individuals and teams. This paper reports research to understand the factors that impact the effectiveness of Cyber Security Incidence Response Teams (CSIRTs). Specifically, a simulation is described that captures the workflow within a CSIRT. The simulation is then demonstrated in a study comparing the differential response time to threats that vary with respect to key characteristics (attack trajectory, targeted asset and perpetrator). It is shown that the results of the simulation correlate with data from the actual incident response times of a professional CSIRT.
22nd Annual Conference on Behavior Representation in Modeling and Simulation, BRiMS 2013 - Co-located with the International Conference on Cognitive Modeling
Modeling agent behaviors in complex task environments requires the agent to be sensitive to complex stimuli such as the positions and actions of varying numbers of other entities. Entity state updates may be received asynchronously rather than on a coordinated clock signal, so the world state must be estimated based on the most recent information available for each entity. The simulation environment is likely to be distributed across several computers over a network. This paper presents the Relational Blackboard (RBB), which is a framework developed to address these needs with clarity and efficiency. The purpose of this paper is to explain the concepts used to represent and process spatio-temporal data in the RBB framework so researchers in related areas can apply the concepts and software to their own problems of interest; detailed description of our own research will be found in other papers. The software is freely available under the BSD open-source license at http://rbb.sandia.gov.
This document reports on the research of Kenneth Letendre, the recipient of a Sandia Graduate Research Fellowship at the University of New Mexico. Warfare is an extreme form of intergroup competition in which individuals make extreme sacrifices for the benefit of their nation or other group to which they belong. Among animals, limited, non-lethal competition is the norm. It is not fully understood what factors lead to warfare. We studied the global variation in the frequency of civil conflict among countries of the world, and its positive association with variation in the intensity of infectious disease. We demonstrated that the burden of human infectious disease importantly predicts the frequency of civil conflict and tested a causal model for this association based on the parasite-stress theory of sociality. We also investigated the organization of social foraging by colonies of harvester ants in the genus Pogonomyrmex, using both field studies and computer models.
This report summarizes research conducted through the Sandia National Laboratories Robust Automated Knowledge Capture Laboratory Directed Research and Development project. The objective of this project was to advance scientific understanding of the influence of individual cognitive attributes on decision making. The project has developed a quantitative model known as RumRunner that has proven effective in predicting the propensity of an individual to shift strategies on the basis of task and experience related parameters. Three separate studies are described which have validated the basic RumRunner model. This work provides a basis for better understanding human decision making in high consequent national security applications, and in particular, the individual characteristics that underlie adaptive thinking.
Training simulators have become increasingly popular tools for instructing humans on performance in complex environments. However, the question of how to provide individualized and scenario-specific assessment and feedback to students remains largely an open question. To maximize training efficiency, new technologies are required that assist instructors in providing individually relevant instruction. Sandia National Laboratories has shown the feasibility of automated performance assessment tools, such as the Sandia-developed Automated Expert Modeling and Student Evaluation (AEMASE) software, through proof-of-concept demonstrations, a pilot study, and an experiment. In the pilot study, the AEMASE system, which automatically assesses student performance based on observed examples of good and bad performance in a given domain, achieved a high degree of agreement with a human grader (89%) in assessing tactical air engagement scenarios. In more recent work, we found that AEMASE achieved a high degree of agreement with human graders (83-99%) for three Navy E-2 domain-relevant performance metrics. The current study provides a rigorous empirical evaluation of the enhanced training effectiveness achievable with this technology. In particular, we assessed whether giving students feedback based on automated metrics would enhance training effectiveness and improve student performance. We trained two groups of employees (differentiated by type of feedback) on a Navy E-2 simulator and assessed their performance on three domain-specific performance metrics. We found that students given feedback via the AEMASE-based debrief tool performed significantly better than students given only instructor feedback on two out of three metrics. Future work will focus on extending these developments for automated assessment of teamwork.
Training simulators have become increasingly popular tools for instructing humans on performance in complex environments. However, the question of how to provide individualized and scenario-specific assessment and feedback to students remains largely an open question. In this work, we follow-up on previous evaluations of the Automated Expert Modeling and Automated Student Evaluation (AEMASE) system, which automatically assesses student performance based on observed examples of good and bad performance in a given domain. The current study provides a rigorous empirical evaluation of the enhanced training effectiveness achievable with this technology. In particular, we found that students given feedback via the AEMASE-based debrief tool performed significantly better than students given only instructor feedback on two out of three domain-specific performance metrics.
The object of the 'Enabling Immersive Simulation for Complex Systems Analysis and Training' LDRD has been to research, design, and engineer a capability to develop simulations which (1) provide a rich, immersive interface for participation by real humans (exploiting existing high-performance game-engine technology wherever possible), and (2) can leverage Sandia's substantial investment in high-fidelity physical and cognitive models implemented in the Umbra simulation framework. We report here on these efforts. First, we describe the integration of Sandia's Umbra modular simulation framework with the open-source Delta3D game engine. Next, we report on Umbra's integration with Sandia's Cognitive Foundry, specifically to provide for learning behaviors for 'virtual teammates' directly from observed human behavior. Finally, we describe the integration of Delta3D with the ABL behavior engine, and report on research into establishing the theoretical framework that will be required to make use of tools like ABL to scale up to increasingly rich and realistic virtual characters.
This project is being conducted by Sandia National Laboratories in support of the DARPA Augmented Cognition program. Work commenced in April of 2002. The objective for the DARPA program is to 'extend, by an order of magnitude or more, the information management capacity of the human-computer warfighter.' Initially, emphasis has been placed on detection of an operator's cognitive state so that systems may adapt accordingly (e.g., adjust information throughput to the operator in response to workload). Work conducted by Sandia focuses on development of technologies to infer an operator's ongoing cognitive processes, with specific emphasis on detecting discrepancies between machine state and an operator's ongoing interpretation of events.