Publications

This report presents a perspective on the role of code comparison activities in verification and validation. We formally define the act of code comparison as the Code Comparison Principle (CCP) and investigate its application in both verification and validation. One of our primary conclusions is that the use of code comparisons for validation is improper and dangerous. We also conclude that while code comparisons may be argued to provide a beneficial component in code verification activities, there are higher quality code verification tasks that should take precedence. Finally, we provide a process for application of the CCP that we believe is minimal for achieving benefit in verification processes.

Several major issues associated with model validation are addressed here. First, we extend the application-based, model validation metric presented in Hills and Trucano (2001) to the Maximum Likelihood approach introduced in Hills and Trucano (2002). This method allows us to use the target application of the code to weigh the measurements made from a validation experiment so that those measurements that are most important for the application are more heavily weighted. Secondly, we further develop the linkage between suites of validation experiments and the target application so that we can (1) provide some measure of coverage of the target application and, (2) evaluate the effect of uncertainty in the measurements and model parameters on application level validation. We provide several examples of this approach based on steady and transient heat conduction, and shock physics applications.

This report describes the results of the FY01 Level 1 Peer Reviews for the Verification and Validation (V&V) Program at Sandia National Laboratories. V&V peer review at Sandia is intended to assess the ASCI (Accelerated Strategic Computing Initiative) code team V&V planning process and execution. The Level 1 Peer Review process is conducted in accordance with the process defined in SAND2000-3099. V&V Plans are developed in accordance with the guidelines defined in SAND2000-3 101. The peer review process and process for improving the Guidelines are necessarily synchronized and form parts of a larger quality improvement process supporting the ASCI V&V program at Sandia. During FY00 a prototype of the process was conducted for two code teams and their V&V Plans and the process and guidelines updated based on the prototype. In FY01, Level 1 Peer Reviews were conducted on an additional eleven code teams and their respective V&V Plans. This report summarizes the results from those peer reviews, including recommendations from the panels that conducted the reviews.

This report presents general concepts in a broadly applicable methodology for validation of Accelerated Strategic Computing Initiative (ASCI) codes for Defense Programs applications at Sandia National Laboratories. The concepts are defined and analyzed within the context of their relative roles in an experimental validation process. Examples of applying the proposed methodology to three existing experimental validation activities are provided in appendices, using an appraisal technique recommended in this report.

The verification and validation (V & V) in computational fluid dynamics was presented. The methods and procedures for assessing V & V were presented. The issues such as code verification versus solution verification, model validation versus solution validation, the distinction between error and uncertainity, conceptual sources of error and uncertainity, and the relationship between validation and prediction was discussed. Methods for determining the accuracy of numerical solutions were presented and the importance of software testing during verification activities were emphasized.

Two major issues associated with model validation are addressed here. First, we present a maximum likelihood approach to define and evaluate a model validation metric. The advantage of this approach is it is more easily applied to nonlinear problems than the methods presented earlier by Hills and Trucano (1999, 2001); the method is based on optimization for which software packages are readily available; and the method can more easily be extended to handle measurement uncertainty and prediction uncertainty with different probability structures. Several examples are presented utilizing this metric. We show conditions under which this approach reduces to the approach developed previously by Hills and Trucano (2001). Secondly, we expand our earlier discussions (Hills and Trucano, 1999, 2001) on the impact of multivariate correlation and the effect of this on model validation metrics. We show that ignoring correlation in multivariate data can lead to misleading results, such as rejecting a good model when sufficient evidence to do so is not available.

This report describes the underlying principles and goals of the Sandia ASCI Verification and Validation Program Validation Metrics Project. It also gives a technical description of two case studies, one in structural dynamics and the other in thermomechanics, that serve to focus the technical work of the project in Fiscal Year 2001.

Abstract not provided.

Abstract not provided.

This report describes the initial definition of the Verification and Validation (V and V) Plan Peer Review Process at Sandia National Laboratories. V and V peer review at Sandia is intended to assess the ASCI code team V and V planning process and execution. Our peer review definition is designed to assess the V and V planning process in terms of the content specified by the Sandia Guidelines for V and V plans. Therefore, the peer review process and process for improving the Guidelines are necessarily synchronized, and form parts of a larger quality improvement process supporting the ASCI V and V program at Sandia.

Verification and validation are the primary means to assess accuracy and reliability in computational simulations. This paper presents an extensive review of the literature in computational validation and develops a number of extensions to existing ideas. We discuss the early work in validation by the operations research, statistics, and CFD communities. The emphasis in our review is to bring together the diverse contributors to validation methodology and procedures. The disadvantages of standard practice of qualitative graphical validation are pointed out and the arguments for and the literature on validation quantification are presented. We discuss the attributes of a beneficial validation experiment hierarchy and then we give an example for a complex system; a hypersonic cruise missile. We present six recommended characteristics of how a validation experiment is designed, executed, and analyzed. Since one of the key features of a validation experiment is a careful experimental uncertainty estimation analysis, we discuss a statistical procedure that has been developed for improving the estimation of experimental uncertainty. One facet of code verification, the estimation of computational error and uncertainty, is discussed in some detail, but we do not address many other important issues in code verification. We argue for the separation of the concepts of error and uncertainty in computational simulations. Error estimation, primarily that due to numerical solution error, is discussed with regard to its importance in validation. In the same vein, we explain the need to move toward nondeterministic simulations in CFD validation, that is, the propagation of input quantity uncertainty in CFD simulations which yield probabilistic output quantities. We discuss the relatively new concept of validation quantification, also referred to as validation metrics. The inadequacy, in our view, of hypothesis testing in computational validation is discussed. We close the paper by presenting our ideas on validation metrics and we apply them to two conceptual examples. © 2000 The American Institute of Aeronautics and Astronautics Inc.