This report presents a perspective on the role of code comparison activities in verification and validation. We formally define the act of code comparison as the Code Comparison Principle (CCP) and investigate its application in both verification and validation. One of our primary conclusions is that the use of code comparisons for validation is improper and dangerous. We also conclude that while code comparisons may be argued to provide a beneficial component in code verification activities, there are higher quality code verification tasks that should take precedence. Finally, we provide a process for application of the CCP that we believe is minimal for achieving benefit in verification processes.
Several major issues associated with model validation are addressed here. First, we extend the application-based, model validation metric presented in Hills and Trucano (2001) to the Maximum Likelihood approach introduced in Hills and Trucano (2002). This method allows us to use the target application of the code to weigh the measurements made from a validation experiment so that those measurements that are most important for the application are more heavily weighted. Secondly, we further develop the linkage between suites of validation experiments and the target application so that we can (1) provide some measure of coverage of the target application and, (2) evaluate the effect of uncertainty in the measurements and model parameters on application level validation. We provide several examples of this approach based on steady and transient heat conduction, and shock physics applications.
This report describes the results of the FY01 Level 1 Peer Reviews for the Verification and Validation (V&V) Program at Sandia National Laboratories. V&V peer review at Sandia is intended to assess the ASCI (Accelerated Strategic Computing Initiative) code team V&V planning process and execution. The Level 1 Peer Review process is conducted in accordance with the process defined in SAND2000-3099. V&V Plans are developed in accordance with the guidelines defined in SAND2000-3 101. The peer review process and process for improving the Guidelines are necessarily synchronized and form parts of a larger quality improvement process supporting the ASCI V&V program at Sandia. During FY00 a prototype of the process was conducted for two code teams and their V&V Plans and the process and guidelines updated based on the prototype. In FY01, Level 1 Peer Reviews were conducted on an additional eleven code teams and their respective V&V Plans. This report summarizes the results from those peer reviews, including recommendations from the panels that conducted the reviews.
This report presents general concepts in a broadly applicable methodology for validation of Accelerated Strategic Computing Initiative (ASCI) codes for Defense Programs applications at Sandia National Laboratories. The concepts are defined and analyzed within the context of their relative roles in an experimental validation process. Examples of applying the proposed methodology to three existing experimental validation activities are provided in appendices, using an appraisal technique recommended in this report.
The verification and validation (V & V) in computational fluid dynamics was presented. The methods and procedures for assessing V & V were presented. The issues such as code verification versus solution verification, model validation versus solution validation, the distinction between error and uncertainity, conceptual sources of error and uncertainity, and the relationship between validation and prediction was discussed. Methods for determining the accuracy of numerical solutions were presented and the importance of software testing during verification activities were emphasized.
Two major issues associated with model validation are addressed here. First, we present a maximum likelihood approach to define and evaluate a model validation metric. The advantage of this approach is it is more easily applied to nonlinear problems than the methods presented earlier by Hills and Trucano (1999, 2001); the method is based on optimization for which software packages are readily available; and the method can more easily be extended to handle measurement uncertainty and prediction uncertainty with different probability structures. Several examples are presented utilizing this metric. We show conditions under which this approach reduces to the approach developed previously by Hills and Trucano (2001). Secondly, we expand our earlier discussions (Hills and Trucano, 1999, 2001) on the impact of multivariate correlation and the effect of this on model validation metrics. We show that ignoring correlation in multivariate data can lead to misleading results, such as rejecting a good model when sufficient evidence to do so is not available.
This report describes the underlying principles and goals of the Sandia ASCI Verification and Validation Program Validation Metrics Project. It also gives a technical description of two case studies, one in structural dynamics and the other in thermomechanics, that serve to focus the technical work of the project in Fiscal Year 2001.
This report describes the initial definition of the Verification and Validation (V and V) Plan Peer Review Process at Sandia National Laboratories. V and V peer review at Sandia is intended to assess the ASCI code team V and V planning process and execution. Our peer review definition is designed to assess the V and V planning process in terms of the content specified by the Sandia Guidelines for V and V plans. Therefore, the peer review process and process for improving the Guidelines are necessarily synchronized, and form parts of a larger quality improvement process supporting the ASCI V and V program at Sandia.
This report summarizes general guidelines for the development of Verification and Validation (V and V) plans for ASCI code projects at Sandia National Laboratories. The main content categories recommended by these guidelines for explicit treatment in Sandia V and V plans are (1) stockpile drivers influencing the code development project (2) the key phenomena to be modeled by the individual code; (3) software verification strategy and test plan; and (4) code validation strategy and test plans. The authors of this document anticipate that the needed content of the V and V plans for the Sandia ASCI codes will evolve as time passes. These needs will be reflected by future versions of this document.
In this combination background and position paper, the authors argue that careful work is needed to develop accurate methods for relating the results of fine-scale numerical simulations of material processes to meaningful values of macroscopic properties for use in constitutive models suitable for finite element solid mechanics simulations. To provide a definite context for this discussion, the problem is couched in terms of the lack of general objective criteria for identifying the size of the representative volume (RV) of a material. The objective of this report is to lay out at least the beginnings of an approach for applying results and methods from statistical physics to develop concepts and tools necessary for determining the RV size, as well as alternatives to RV volume-averaging for situations in which the RV is unmanageably large. The background necessary to understand the pertinent issues and statistical physics concepts is presented.
A tutorial is presented discussing the basic issues associated with propagation of uncertainty analysis and statistical validation of engineering and scientific models. The propagation of uncertainty tutorial illustrates the use of the sensitivity method and the Monte Carlo method to evaluate the uncertainty in predictions for linear and nonlinear models. Four example applications are presented; a linear model, a model for the behavior of a damped spring-mass system, a transient thermal conduction model, and a nonlinear transient convective-diffusive model based on Burger's equation. Correlated and uncorrelated model input parameters are considered. The model validation tutorial builds on the material presented in the propagation of uncertainty tutoriaI and uses the damp spring-mass system as the example application. The validation tutorial illustrates several concepts associated with the application of statistical inference to test model predictions against experimental observations. Several validation methods are presented including error band based, multivariate, sum of squares of residuals, and optimization methods. After completion of the tutorial, a survey of statistical model validation literature is presented and recommendations for future work are made.
In this paper, we report details of our computational study of two shock wave physics experiments performed on the Sandia Z machine in 1998. The novelty of these particular experiments is that they represent the first successful appli- cation of VISAR interferometry to diagnose shock waves generated in experi- mental payloads by the primary X-ray pulse of the machine. We use the Sandia shock-wave physics code ALEGRA to perform the simulations reported in this study. Our simulations are found to be in fair agreement with the time-resolved VISAR experimental data. However, there are also interesting and important discrepancies. We speculate as to future use of time-resolved shock wave data to diagnose details of the Z machine X-ray pulse in the future.
The US Department of Energy`s (DOE) Accelerated Strategic Computing Initiative (ASCI) program calls for the development of high end computing and advanced application simulations as one component of a program to eliminate reliance upon nuclear testing in the US nuclear weapons program. This paper presents results from the ASCI program`s examination of needs for focused validation and verification (V and V). These V and V activities will ensure that 100 TeraOP-scale ASCI simulation code development projects apply the appropriate means to achieve high confidence in the use of simulations for stockpile assessment and certification. The authors begin with an examination of the roles for model development and validation in the traditional scientific method. The traditional view is that the scientific method has two foundations, experimental and theoretical. While the traditional scientific method does not acknowledge the role for computing and simulation, this examination establishes a foundation for the extension of the traditional processes to include verification and scientific software development that results in the notional framework known as Sargent`s Framework. This framework elucidates the relationships between the processes of scientific model development, computational model verification and simulation validation. This paper presents a discussion of the methodologies and practices that the ASCI program will use to establish confidence in large-scale scientific simulations. While the effort for a focused program in V and V is just getting started, the ASCI program has been underway for a couple of years. The authors discuss some V and V activities and preliminary results from the ALEGRA simulation code that is under development for ASCI. The breadth of physical phenomena and the advanced computational algorithms that are employed by ALEGRA make it a subject for V and V that should typify what is required for many ASCI simulations.
Detailed computational modeling of laser surgery requires treatment of the photoablation of human tissue by high intensity pulses of laser light and the subsequent thermomechanical response of the tissue. Three distinct physical regimes must be considered to accomplish this: (1) the immediate absorption of the laser pulse by the tissue and following tissue ablation, which is dependent upon tissue light absorption characteristics; (2) the near field thermal and mechanical response of the tissue to this laser pulse, and (3) the potential far field (and longer time) mechanical response of witness tissue. Both (2) and (3) are dependent upon accurate constitutive descriptions of the tissue. We will briefly review tissue absorptivity and mechanical behavior, with an emphasis on dynamic loads characteristic of the photoablation process. In this paper our focus will center on the requirements of numerical modeling and the uncertainties of mechanical tissue behavior under photoablation. We will also discuss potential contributions that computational simulations can make in the design of surgical protocols which utilize lasers, for example, in assessing the potential for collateral mechanical damage by laser pulses.
The purpose of this report is to summarize the status of computational analysis of hypervelocity impact lethality in relatively nontechnical terms from the perspective of the author. It is not intended to be a review of the technical literature on the problems of concern. The discussion is focused by concentrating on two phenomenology areas which are of particular concern in computational impact studies. First, the material`s equation of state, specifically the treatment of expanded states of metals undergoing shock vaporization, is discussed. Second, the process of dynamic fragmentation is addressed. In both cases, the context of the discussion deals with inaccuracies and difficulties associated with numerical hypervelocity impact simulations. Laboratory experimental capabilities in hypervelocity impact for impact velocities greater than 10.0 km/s are becoming increasingly viable. This paper also gives recommendations for experimental thrusts which utilize these capabilities that will help to resolve the uncertainties in the numerical lethality studies that are pointed out in the present report.
The following research is proposed: (1) Perform four to six new time-resolved lethality experiments on cadmium at projectile velocities 7 km/s, greater than that of our previous work at 5.2 km/s. The increased velocity will substantially increase the mass percentage of vaporization. Through simulation of these experiments, computer codes can be validated for hypervelocity impact when vaporization occurs. (2) Perform an additional four to six 1-D lethality experiments on zinc. This material undergoes substantial vaporization under the experimental conditions, and would expand our knowledge about the sensitivity of KEW lethality to shock-induced vaporization. (3) Perform two to four ballistic impact experiments with the ballistics gun at Sandia to provide a full validation of computational capabilities used in simulating the 1-D experiments. These experiments would probe phenomenology that is similar to the 1-D experiments, but with multi-dimensional debris propagation. This will extend the confidence in computer modeling to conditions analagous to KEW lethality applications.