Archive of earlier Uncertainty Quantification/Verification & Validation Seminar Series—Seminars 12–16

During this time of declining budgets, we must optimize the few investment dollars available to maximize value to the weapon system. The lack of a profit motive means that Net Present Value (the common measure of value in industry) is not an applicable measure for Sandia. Recognizing that the value of testing lies in improved confidence that the weapon system will work, this approach leverages quantified margins and uncertainty around the reliability estimates to quantify the reduction in risk that Sandia gains by executing IDL projects (different component testing projects). By doing so, we can measure our improved confidence and thus the relative value gained by executing each test program. We demonstrate this approach with a simple example of using risk based valuation within a decision analysis context to establish the relative value of proposed IDL projects."

This tutorial will present an introduction to model validation and address how to develop criteria to assess model validation. It will also cover uncertainty quantification from the perspective of probabilistic analysis as well as epistemic analysis. All of these tutorial talks are based around a real application at Sandia involving a finite element structural dynamics model of a three-layered conical shell structure, with an emphasis on understanding the uncertainty in the material properties. The schedule of the tutorial is as follows, with more detail on each of the talks listed below. You are welcome to attend any or all of the talks. The authors are planning to present this tutorial at the Society for Experimental Mechanics IMAC conference in February.

1:00. Introduction to Model Validation, Tom Paez

1:30. Developing Adequacy Criterion for Model Validation based on Requirements, Randy Mayes

2:00. Probabilistic Methods in Model Validation, Tom Paez

2:30. Epistemic Uncertainty Quantification Tutorial, Laura Swiler

1:00. Introduction to Model Validation, Tom Paez
The discipline of mathematical model validation is increasing in importance as the value of accurate models of physical systems increases. The fundamental activity of model validation is the comparison of predictions from a mathematical model of a system to the measured behavior of the system. This discussion introduces some preliminary elements of model validation including (1) specification of a validation hierarchy and definition of validation experiments, (2) specification of modeling assumptions and definition of the mathematical and computational models, (3) specification of response measures of interest, (4) specification of validation metrics and the means used to compare model predictions to experimental results, (5) specification of the domain of comparison, (6) specification of calibration experiments, and (7) specification of adequacy criteria. These specifications precede (1) the performance of calibration experiments, (2) the production of model predictions, (3) the performance of validation experiments, and (4) the performance of validation comparisons and judgment of model adequacy.

1:30. Developing Adequacy Criterion for Model Validation based on Requirements, Randy Mayes
Model validation assesses the usefulness of a model for its intended purpose. It is often difficult to develop a quantitative measure to determine if the model is acceptable for the intended purpose. In this paper, it is assumed that the model will have systematic differences from the validation test data. The ultimate requirement is that the model be good enough to provide guidance for an engineering decision that must be made. It is then useful. A rationale must be developed that relates one or more response features to a requirement that is tied to the decision that will be made with the model predictions. At that point, one or more adequacy criteria need to be set in advance of the validation prediction of the response feature. One common mistake is to select a feature that is common in the industry, but cannot be easily related to the requirement for the model. Another common mistake is setting the adequacy criteria too stringently so that the model fails the validation even though it is useful for guiding the decision. Minimizing the number of response features is desirable for making the validation decision more straightforward. The rationale for selection of features and adequacy criteria should be agreed upon by the decision maker, analyst, test engineer and other stakeholders.

2:00. Probabilistic Methods in Model Validation, Tom Paez
Extensive experimentation over the past decade has shown that fabricated physical systems that are intended to be identical, and are nominally identical, in fact, differ from one another, and sometimes substantially. This fact makes it difficult to validate a mathematical model for any system and results in the requirement to characterize physical system behavior using the tools of uncertainty quantification. Further, because of the existence of system, component, and material uncertainty the mathematical models of these elements sometimes seek to reflect the uncertainty. This presentation introduces some of the methods of probability and statistics, and shows how they can be applied in engineering modeling and data analysis. The ideas of randomness and some basic means for measuring and modeling it are presented. The ideas of random experiment, random variable, mean, variance and standard deviation, and probability distribution are introduced. The ideas are introduced in the framework of a practical, yet simple, example; measured data are included.

2:30. Epistemic Uncertainty Quantification Tutorial, Laura Swiler
This paper presents a basic tutorial on epistemic uncertainty quantification methods. Epistemic uncertainty, characterizing lack-of-knowledge, is often prevalent in engineering applications. However, the methods we have for analyzing and propagating epistemic uncertainty are not as nearly widely used or well-understood as methods to propagate aleatory uncertainty (e.g. inherent variability characterized by probability distributions). We examine three methods used in propagating epistemic uncertainties: interval analysis, Dempster-Shafer evidence theory, and second-order probability. We demonstrate examples of their use on a problem in structural dynamics.

Nuclear weapons testing is a key application area at Sandia. Weapons are complex systems of interconnected components for which reliability is of paramount importance, and the processes for estimating this reliability quantitatively are extraordinarily complicated. While these estimation processes use analytical methods that incorporate data acquired through an array of testing procedures, there is significant pressure to reduce the costs of testing while maintaining the integrity of the stockpile. Moreover, a reliability estimate is just that, an estimate, particularly for systems for which it is impractical, or impossible, to perform sufficient classical tests, to ensure a specified level of confidence. In these cases, it is relevant to establish a formal process for assessing the uncertainty in the estimate itself. Both of these aspects motivate our attempt to investigate alternatives to our current practices in weapons testing and reliability assessment.

In this talk, we describe novel mathematical strategies, based on structured probabilistic models, for testing our systems. We propose to use these strategies to assess the impact of various combinations of subsystem- and component-level tests on overall system reliability, and its associated uncertainty due to epistemological realities. We will also consider the problem of predicting future reliability given time-dependent data. The Bayesian approach of Martz-Waller will first be extended to testing regimes where continuous data, e.g., voltage, are collected in addition to that collected in traditional, and simpler,“pass-fail” tests. This approach will be extended further to accommodate time-dependent systems; our goal is to handle predictions in an environment where aging may be important. Finally, we will extend this approach to more general network-class applications for which certain components may exist in, or directly affect, multiple systems. Exact, or approximate, inference algorithms will allow probability distributions characterizing reliability to be updated through and across systems. This ability is crucial for estimating the uncertainty in an overall system reliability estimate. Our construction will, for the first time, enable the formulation of optimization questions concerning the best testing strategies and the relative importance of each individual test on the overall reliability and uncertainty estimates.
We present some numerical results and suggest possible extensions to other applications. ipsum dolor sit amet, contuer adiping elit, sed diam nonummy nibh euismod.

Lorem ipsum dolor sit amed sed diam nonummy nibh euismod tincidunt ut. Duis autem vel eum iriure et accumsan et odio qui blandit praesent luptatum augue duis dolore te feugait nulla facilisi.

Two classical verification problems from shock hydrodynamics are adapted for verification in the context of ideal magnetohydrodynamics (MHD) by introducing strong transverse magnetic fields, and simulated using the finite element Lagrange-remap MHD code ALEGRA for purposes of rigorous code verification. The concern in these verification tests is that inconsistencies related to energy advection are inherent in Lagrange-remap formulations for MHD, such that conservation of the kinetic and magnetic components of the energy may not be maintained. Hence, total energy conservation may also not be maintained. MHD shock propagation may therefore not be treated consistently in Lagrange-remap schemes, as errors in energy conservation are known to result in unphysical shock wave speeds and post-shock states.

That kinetic energy is not conserved in Lagrange-remap schemes is well known, and the correction of DeBar has been shown to eliminate the resulting errors. Here, the consequences of the failure to conserve magnetic energy are revealed using order verification in the two magnetized shock-hydrodynamics problems. Further, a magnetic analog to the DeBar correction is proposed and its accuracy evaluated using this verification testbed. Results indicate that only when the total energy is conserved, by implementing both the kinetic and magnetic components of the DeBar correction, can simulations in Lagrange-remap formulation capture MHD shock propagation accurately. Additional insight is provided by the verification results, regarding the implementation of the DeBar correction and the advection scheme.

Impact of Coding Mistakes on Numerical Error and Uncertainty in Solutions to PDEs, by Patrick M. Knupp, Curtis C. Ober*, Ryan B. Bond

We investigated one source of uncertainty, the numerical error (NE), which is the difference between the numerical solution and the exact solution to the PDE. NE arises from four sources of error within a numerical calculation: (1) discretization error (DE), (2) roundoff error (RE), (3) incomplete iterative convergence error (IICE), and (4) implementation correctness error (ICE). ICE arises from the presence of coding mistakes (bugs) that prevent the correct numerical solution from being computed.

The main purpose of this study was to obtain some insight into the magnitude and effects of coding mistakes (ICE) on the numerical error, sensitivities and uncertainties in the solutions to PDEs. A simple 1D PDE was used in the investigation to circumvent difficulties in using large complex applications, to make use of an exact solution, and to better relate the impact of ICE to the quantities of interest. Using simple 'typo'-type mistakes, this study illustrates many common problems caused by coding mistakes and how they effect the numerical error and uncertainty. From solutions that blow up, to converging to an incorrect answer, the simple-model problem demonstrates how insidious some of the coding mistakes can be. Leading to the concern that it can be worse for our complex-physics applications!