While ML classifiers are widespread, output is often not part of a follow-on decision-making process because of lack of uncertainty quantification. Through this project, the team developed decision analysis methods that combined uncertainty estimates for ML predictions with a domain-specific model of error costs. In the project, they explicitly weighed whether ML models under evaluation were qualified to make any prediction by producing general algorithms that minimized prediction error costs by validating these algorithms through their demonstrations on cyber security and image analysis cases.
The developed and trained ML classifier ultimately provided a framework for: (1) quantifying and propagating uncertainty in ML classifiers; (2) formally linking ML outputs with the decision-making process; and (3) making optimal decisions for classification under uncertainty with single or multiple objectives. Methods developed through this project are currently being incorporated into applications that impact national security domains by directly addressing questions of automated decision.
Sandia researchers linked to work
- Michael Darling
- Justin Doak
Sponsored by
Associated Pubications
Justin Doak, Michael Darling, (2022). Preliminary Results for Using Uncertainty and Out-of-distribution Detection to Identify Unreliable Predictions https://doi.org/10.2172/1899654 Publication ID: 80438
Richard Field, Michael Darling, (2022). A Decision Theoretic Approach To Optimizing Machine Learning Decisions with Prediction Uncertainty https://doi.org/10.2172/1899419 Publication ID: 80436
Michelle Leger, Michael Darling, Stephen Jones, Laura Matzen, David Stracuzzi, Andrew Wilson, Denis Bueno, Matthew Christentsen, Melissa Ginaldi, David Hannasch, Scott Heidbrink, Breannan Howell, Chris Leger, Geoffrey Reedy, Alisa Rogers, Jack Williams, (2021). Exploring Explicit Uncertainty for Binary Analysis (EUBA) https://doi.org/10.2172/1832314 Publication ID: 76829
Michael Darling, Justin Doak, Richard Field, Mark Smith, (2021). Optimizing Machine Learning Decisions with Prediction Uncertainty https://doi.org/10.2172/1888406 Publication ID: 79403
Zachary Smith, Michael Darling, David Stracuzzi, Justin Doak, Mark Smith, J. Bickel, (2021). Optimizing Machine Learning Predictions with Prediction Uncertainty https://doi.org/10.2172/1860307 Publication ID: 77800
Justin Doak, (2020). I could see your lips move https://www.osti.gov/servlets/purl/1804992 Publication ID: 73931
Justin Doak, Michael Smith, Joey Ingram, (2020). Self-Updating Models with Error Remediation https://doi.org/10.1117/12.2563843 Publication ID: 73520
Nancy Hayden, Jason Reinhardt, Mallory Stewart, Justin Doak, Thushra Gunda, Kelsey Abel, (2020). Artificial Intelligence and Autonomy in Space: Balancing Risks of Unintended Escalation https://www.osti.gov/servlets/purl/1780585 Publication ID: 73350
Michael Darling, Don Hush, David Stracuzzi, (2020). A Novel Measure of Uncertainty For Machine Learning Predictions https://www.osti.gov/servlets/purl/1768156 Publication ID: 72788
Michael Darling, (2019). Using Uncertainty To Interpret Supervised Machine Learning Predictions https://www.osti.gov/biblio/1592876 Publication ID: 66905
Justin Doak, Joey Ingram, Michael Smith, Craig Vineyard, (2019). Self-updating Models with Error Remediation for Bandwidth-constrained Environments https://www.osti.gov/servlets/purl/1641249 Publication ID: 69923
Justin Doak, (2019). I could see your lips move https://www.osti.gov/servlets/purl/1644871 Publication ID: 67996
Justin Doak, Joey Ingram, Samuel Mulder, John Naegle, Jonathan Cox, James Aimone, Kevin DIxon, Conrad James, David Follett, (2018). Tracking Cyber Adversaries with Adaptive Indicators of Compromise Proceedings – 2017 International Conference on Computational Science and Computational Intelligence, CSCI 2017 https://doi.org/10.1109/CSCI.2017.2 Publication ID: 54611
David Stracuzzi, Michael Darling, Matthew Peterson, Maximillian Chen, (2018). Quantifying Uncertainty to Improve Decision Making in Machine Learning https://doi.org/10.2172/1481629 Publication ID: 59354
Maximillian Chen, Michael Darling, David Stracuzzi, (2018). Preliminary Results on Applying Nonparametric Clustering and Bayesian Consensus Clustering Methods to Multimodal Data https://doi.org/10.2172/1475256 Publication ID: 59212
Michael Smith, Joey Ingram, Christopher Lamb, Timothy Draelos, Justin Doak, James Aimone, Conrad James, (2018). Dynamic Analysis of Executables to Detect and Characterize Malware https://doi.org/10.1109/ICMLA.2018.00011 Publication ID: 59291
Maximillian Chen, David Stracuzzi, Michael Darling, (2018). A Mathematical Framework for Uncertainty Quantification in Multimodal Image Analysis via Probabilistic Clustering Models https://www.osti.gov/servlets/purl/1591721 Publication ID: 63998
Maximillian Chen, David Stracuzzi, Michael Darling, (2018). A Mathematical Framework for Uncertainty Quantification in Multimodal Image Analysis via Probabilistic Clustering Models https://www.osti.gov/servlets/purl/1575046 Publication ID: 63512
Justin Doak, Joshua Coon, Rolando Fernandez, Mark Louie, Theodore Stangebye, (2018). Capsule Networks: Capturing Presence and Orientation of Representations https://www.osti.gov/servlets/purl/1573310 Publication ID: 63439
Maximillian Chen, Michael Darling, Charlie Vollmer, Matthew Peterson, David Stracuzzi, (2018). Using Uncertainty to Understand Machine Learning Models and Decisions https://www.osti.gov/servlets/purl/1526113 Publication ID: 62451
David Stracuzzi, Michael Darling, Maximillian Chen, Matthew Peterson, (2018). Data-Driven Uncertainty Quantification for Multi-Sensor Analytics https://www.osti.gov/servlets/purl/1507484 Publication ID: 61468
Michael Darling, David Stracuzzi, Maximillian Chen, (2018). Uncertainty Propagation In Multilayer Analysis https://www.osti.gov/servlets/purl/1498629 Publication ID: 60943
Michael Darling, George Luger, Thomas Jones, Matthew Denman, Katrina Groth, (2018). Intelligent Modeling for Nuclear Power Plant Accident Management International Journal on Artificial Intelligence Tools https://doi.org/10.1142/S0218213018500033 Publication ID: 58554
Maximillian Chen, Michael Darling, David Stracuzzi, (2018). Multimodal Image Analysis and Uncertainty Quantification via Nonparametric Probabilistic Clustering https://www.osti.gov/servlets/purl/1498444 Publication ID: 60902
David Stracuzzi, Michael Darling, Maximillian Chen, Matthew Peterson, (2018). Data-driven uncertainty quantification for multisensor analytics Proceedings of SPIE – The International Society for Optical Engineering https://doi.org/10.1117/12.2304921 Publication ID: 61586
Joey Ingram, Timothy Draelos, Meghan Sahakian, Justin Doak, (2017). Temporal Cyber Attack Detection https://doi.org/10.2172/1409921 Publication ID: 54278
Maximillian Chen, David Stracuzzi, Michael Darling, Matthew Peterson, (2017). Establishing Uniform Image Segmentation Ground Truth Protocols for Uncertainty Quantification and Improved Model Evaluation https://www.osti.gov/servlets/purl/1511807 Publication ID: 54356
Justin Doak, Justin Doak, Joey Ingram, Joey Ingram, Samuel Mulder, Samuel Mulder, John Naegle, John Naegle, Jonathan Cox, Jonathan Cox, James Aimone, James Aimone, Kevin DIxon, Kevin DIxon, Conrad James, Conrad James, David Follet, David Follet, (2017). Tracking Cyber Adversaries with Adaptive Indicators of Compromise https://doi.org/10.1109/CSCI.2017.2 Publication ID: 54191
David Stracuzzi, Maximillian Chen, Michael Darling, Matthew Peterson, (2017). From Data to Decisions: Placing Machine Learning Challenges In Context https://www.osti.gov/servlets/purl/1484938 Publication ID: 54484
David Stracuzzi, Maximillian Chen, Michael Darling, Stephen Dauphin, Matthew Peterson, Charlie Vollmer, (2017). Data-Driven Uncertainty Quantification for Remote Sensing https://www.osti.gov/servlets/purl/1457960 Publication ID: 56281
David Stracuzzi, Maximillian Chen, Michael Darling, Stephen Dauphin, Matthew Peterson, Charlie Vollmer, Christopher Young, (2016). Uncertainty in Data Analytics https://www.osti.gov/servlets/purl/1413588 Publication ID: 48176
Michael Darling, Katrina Groth, (2016). A Dynamic Bayesian Network for Diagnosing Nuclear Power Plant Accidents https://www.osti.gov/servlets/purl/1375590 Publication ID: 51647
Justin Doak, (2016). Cyber Training Curriculum https://www.osti.gov/servlets/purl/1514625 Publication ID: 51375
Michael Darling, Katrina Groth, Matthew Denman, Thomas Jones, George Luger, (2016). A Dynamic Bayesian Network for Diagnosing Nuclear Power Plant Accidents https://www.osti.gov/servlets/purl/1367135 Publication ID: 49938
Michael Bierma, Justin Doak, Corey Hudson, (2016). Learning to Rank for Alert Triage https://doi.org/10.1109/THS.2016.7568907 Publication ID: 49871
Michael Bierma, Justin Doak, Corey Hudson, (2016). Learning to Rank for Alert Triage https://doi.org/10.1109/THS.2016.7568907 Publication ID: 49371
Thomas Jones, Michael Darling, Katrina Groth, Matthew Denman, George Luger, (2016). A dynamic Bayesian network for diagnosing nuclear power plant accidents Proceedings of the 29th International Florida Artificial Intelligence Research Society Conference, FLAIRS 2016 https://www.osti.gov/servlets/purl/1345095 Publication ID: 48503
Joshua Johnson, Justin Doak, Joey Ingram, Jeffery Shelburg, (2013). Active Learning for Alert Triage https://www.osti.gov/servlets/purl/1121155 Publication ID: 31722
Justin Doak, Joey Ingram, Joshua Johnson, Jeffery Shelburg, (2013). Active Learning for Alert Triage https://www.osti.gov/servlets/purl/1115032 Publication ID: 35198
Roger Suppona, Andrew Wilson, Justin Doak, (2013). Can we identify spear phishing targets before the email is sent? https://www.osti.gov/servlets/purl/1115637 Publication ID: 33496
Jason Haas, Justin Doak, Sean Crosby, Ryan Helinski, Christopher Lamb, (2013). Dynamic defense workshop : https://doi.org/10.2172/1093703 Publication ID: 32106
Roger Suppona, Justin Doak, (2012). LDRD Annual Report blurb for Jeremy Wendt’s LDRD https://www.osti.gov/servlets/purl/1658291 Publication ID: 30318
Jason Haas, Justin Doak, Jason Hamlet, (2012). Machine-Oriented Biometrics and Cocooning for Dynamic Network Defense https://www.osti.gov/servlets/purl/1116590 Publication ID: 29628
Justin Doak, Andrew Wilson, Roger Suppona, (2012). Identifying Dynamic Patterns in Network Traffic to Predict and Mitigate Cyberattacks https://www.osti.gov/servlets/purl/1658713 Publication ID: 29028
Showing Results.