As machine learning (ML) models are deployed into an ever-diversifying set of application spaces, ranging from self-driving cars to cybersecurity to climate modeling, the need to carefully evaluate model credibility becomes increasingly important. Uncertainty quantification (UQ) provides important information about the ability of a learned model to make sound predictions, often with respect to individual test cases. However, most UQ methods for ML are themselves data-driven and therefore susceptible to the same knowledge gaps as the models themselves. Specifically, UQ helps to identify points near decision boundaries where the models fit the data poorly, yet predictions can score as certain for points that are under-represented by the training data and thus out-of-distribution (OOD). One method for evaluating the quality of both ML models and their associated uncertainty estimates is out-of-distribution detection (OODD). We combine OODD with UQ to provide insights into the reliability of the individual predictions made by an ML model.
While the use of machine learning (ML) classifiers is widespread, their output is often not part of any follow-on decision-making process. To illustrate, consider the scenario where we have developed and trained an ML classifier to find malicious URL links. In this scenario, network administrators must decide whether to allow a computer user to visit a particular website, or to instead block access because the site is deemed malicious. It would be very beneficial if decisions such as these could be made automatically using a trained ML classifier. Unfortunately, due to a variety of reasons discussed herein, the output from these classifiers can be uncertain, rendering downstream decisions difficult. Herein, we provide a framework for: (1) quantifying and propagating uncertainty in ML classifiers; (2) formally linking ML outputs with the decision-making process; and (3) making optimal decisions for classification under uncertainty with single or multiple objectives.
Reverse engineering (RE) analysts struggle to address critical questions about the safety of binary code accurately and promptly, and their supporting program analysis tools are simply wrong sometimes. The analysis tools have to approximate in order to provide any information at all, but this means that they introduce uncertainty into their results. And those uncertainties chain from analysis to analysis. We hypothesize that exposing sources, impacts, and control of uncertainty to human binary analysts will allow the analysts to approach their hardest problems with high-powered analytic techniques that they know when to trust. Combining expertise in binary analysis algorithms, human cognition, uncertainty quantification, verification and validation, and visualization, we pursue research that should benefit binary software analysis efforts across the board. We find a strong analogy between RE and exploratory data analysis (EDA); we begin to characterize sources and types of uncertainty found in practice in RE (both in the process and in supporting analyses); we explore a domain-specific focus on uncertainty in pointer analysis, showing that more precise models do help analysts answer small information flow questions faster and more accurately; and we test a general population with domain-general sudoku problems, showing that adding "knobs" to an analysis does not significantly slow down performance. This document describes our explorations in uncertainty in binary analysis.
Data-driven modeling, including machine learning methods, continue to play an increas- ing role in society. Data-driven methods impact decision making for applications ranging from everyday determinations about which news people see and control of self-driving cars to high-consequence national security situations related to cyber security and analysis of nuclear weapons reliability. Although modern machine learning methods have made great strides in model induction and show excellent performance in a broad variety of complex domains, uncertainty remains an inherent aspect of any data-driven model. In this report, we provide an update to the preliminary results on uncertainty quantifi- cation for machine learning presented in SAND2017-6776. Specifically, we improve upon the general problem definition and expand upon the experiments conducted for the earlier re- port. Most importantly, we summarize key lessons learned about how and when uncertainty quantification can inform decision making and provide valuable insights into the quality of learned models and potential improvements to them. Acknowledgements The authors thank Kristina Czuchlewski, John Feddema, Todd Jones, Chris Young, Rudy Garcia, Rich Field, Ann Speed, Randy Brost, Stephen Dauphin, and countless others for providing helpful discussion and comments throughout the life of this project. This work was funded by the Sandia National Laboratories Laboratory Directed Research and Development (LDRD) program.
In this report, we present preliminary research into nonparametric clustering methods for multi-source imagery data and quantifying the performance of these models. In many domain areas, data sets do not necessarily follow well-defined and well-known probability distributions, such as the normal, gamma, and exponential. This is especially true when combining data from multiple sources describing a common set of objects (which we call multimodal analysis), where the data in each source can follow different distributions and need to be analyzed in conjunction with one another. This necessitates nonparametric den- sity estimation methods, which allow the data to better dictate the distribution of the data. One prominent example of multimodal analysis is multimodal image analysis, when we an- alyze multiple images taken using different radar systems of the same scene of interest. We develop uncertainty analysis methods, which are inherent in the use of probabilistic models but often not taken advance of, to assess the performance of probabilistic clustering methods used for analyzing multimodal images. This added information helps assess model perfor- mance and how much trust decision-makers should have in the obtained analysis results. The developed methods illustrate some ways in which uncertainty can inform decisions that arise when designing and using machine learning models. Acknowledgements This work was funded by the Sandia National Laboratories Laboratory Directed Research and Development (LDRD) program.
This paper explores the viability of using counterfactual reasoning for impact analyses when understanding and responding to "beyond-design-basis" nuclear power plant accidents. Currently, when a severe nuclear power plant accident occurs, plant operators rely on Severe Accident Management Guidelines. However, the current guidelines are limited in scope and depth: for certain types of accidents, plant operators would have to work to mitigate the damage with limited experience and guidance for the particular situation. We aim to fill the need for comprehensive accident support by using a dynamic Bayesian network to aid in the diagnosis of a nuclear reactor's state and to analyze the impact of possible response measures. The dynamic Bayesian network, DBN, offers an expressive representation of the components and relationships that make up a complex causal system. For this reason, and for its tractable reasoning, the DBN supports a functional model for the intricate operations of nuclear power plants. In this domain, it is also pertinent that a Bayesian network can be composed of both probabilistic and knowledge-based components. Though probabilities can be calculated from simulated models, the structure of the network, as well as the value of some parameters, must be assigned by human experts. Since dynamic Bayesian network-based systems are capable of running better-than-real-time situation analyses, they can support both current event and alternate scenario impact analyses.
We discuss uncertainty quantification in multisensor data integration and analysis, including estimation methods and the role of uncertainty in decision making and trust in automated analytics. The challenges associated with automatically aggregating information across multiple images, identifying subtle contextual cues, and detecting small changes in noisy activity patterns are well-established in the intelligence, surveillance, and reconnaissance (ISR) community. In practice, such questions cannot be adequately addressed with discrete counting, hard classifications, or yes/no answers. For a variety of reasons ranging from data quality to modeling assumptions to inadequate definitions of what constitutes "interesting" activity, variability is inherent in the output of automated analytics, yet it is rarely reported. Consideration of these uncertainties can provide nuance to automated analyses and engender trust in their results. In this work, we assert the importance of uncertainty quantification for automated data analytics and outline a research agenda. We begin by defining uncertainty in the context of machine learning and statistical data analysis, identify its sources, and motivate the importance and impact of its quantification. We then illustrate these issues and discuss methods for data-driven uncertainty quantification in the context of a multi-source image analysis example. We conclude by identifying several specific research issues and by discussing the potential long-term implications of uncertainty quantification for data analytics, including sensor tasking and analyst trust in automated analytics.
When a severe nuclear power plant accident occurs, plant operators rely on Severe Accident Management Guidelines (SAMGs). However, current SAMGs are limited in scope and depth. The plant operators must work to mitigate the accident with limited experience and guidance for the situation. The SMART (Safely Managing Accidental Reactor Transients) procedures framework aims to fill the need for detailed guidance by creating a comprehensive probabilistic model, using a Dynamic Bayesian Network, to aid in the diagnosis of the reactor's state. In this paper, we explore the viability of the proposed SMART proceedures approach by building a prototype Bayesian network that allows tor the diagnosis of two types of accidents based on a comprehensive data set. We use Kullback-Leibler (K-L) divergence to gauge the relative importance of each of the plant's parameters. We compare accuracy and F-score measures across four different Bayesian networks: a baseline network that ignores observation variables, a network that ignores data from the observation variable with the highest K-L score, a network that ignores data from the variable with the lowest K-L score, and finally a network that includes all observation variable data. We conclude with an interpretation of these results for SMART procedures.