Validation & Verification Program Plan

Establish confidence in the simulations supporting the Stockpile Stewardship Program through systematic demonstration and documentation of the predictive capability of the codes and their underlying models.

Provide the basis by which computational simulations are evaluated and assessed.

As a result of the United States’ intention to pursue a "zero yield" Comprehensive Test Ban Treaty, the Department of Energy has undertaken the Stockpile Stewardship Program (SSP) to ensure confidence in the safety, performance, and reliability of the US nuclear stockpile. A greater reliance on computational modeling and simulation is called for as a cornerstone of the SSP. Because simulation plays a key role for the SSP, this focused validation and verification (V&V) program is essential.

This plan outlines such a program for ensuring that computer code projects apply the appropriate means to achieve high confidence in the use of simulations for stockpile assessment and certification. Specific validation and verification activities are described in detail in the Stockpile Computing and Accelerated Strategic Computing Initiative (ASCI) code project implementation plans.

This plan emphasizes validation and verification efforts for large, 100 TeraOP-scale ASCI code projects, since these clearly present the greatest V&V challenges. Most validation and verification resources will be applied to these large codes, but it is understood that the program plan encompasses V&V needs for all SSP code projects.

Background/Philosophy

The DOE Defense Programs SSP Implementation Plan (the Green Book) calls for judgment-based confidence as a necessary requirement for less reliance upon nuclear testing in the U.S. nuclear weapons program. A simulation is considered predictive if it represents a major means of providing information to a weapons expert in all SSP applications relevant to the simulation, and the primary (or only) means in specific critical stockpile situations. The goal for this V&V program is to develop simulations that are predictive in this sense.

Discussion of Terms

Defense Programs should insure common definitions across all of the code development efforts and should endeavor to standardize usage with that of the larger community, in particular DOD customers. The following definitions are consistent with those of the larger community:

Validation - the process of determining the degree to which a computer simulation is an accurate representation of the real world.

Verification - the process of determining that a computer simulation correctly represents the conceptual model and its solution.

"Verification and Validation" or
"Validation and Verification"?

It is common to see the former phrase because chronologically simulations are verified before they are validated. However, because in this program the most visible process is validation, and a key element of this program is to coordinate with other programs to obtain needed validation data, we have chosen to use the latter phrase.

The traditional scientific method is based on a process of observation, hypothesis development, experimental design, hypothesis test, and iterative improvement. For hundreds of years this framework for science had two foundations, experimental and theoretical. Since the development of computers in the 1940's for the Manhattan Project, we now recognize a third foundation, computational science.

The processes of validation and verification deal respectively with the interfaces between computational simulations and experimental observations, and computer models and theoretical models. To illustrate the interactions between these three foundations of science it is useful to examine the scientific framework shown in Figure 1. This figure is based on similar diagrams from the Society for Computer Simulation and the IEEE Computer Society. In this framework there are three main objects: physical reality, conceptual models, and computer simulations, which are respectively, the subject or product of the experimental, theoretical, and computational sciences. The interface processes between these objects are shown in the outer ring, and they are supported by the activities shown in the inner boxes along the dashed lines. Finally all of these objects, processes, and activities are supported with a common need for data validation.

framework

Figure 1. A framework for the new scientific method. This diagram shows how computational science interacts with the traditional activities of experimental and theoretical science.

This background discussion on the new scientific method provides a context for the activities that support the validation and verification processes. The specific validation and verification activities described in this plan are intended to provide:

Impact on the SSP: Validation requirements for improved theory, models and material properties will define appropriate experiments and diagnostics, measurements carried out as part of enhanced surveillance activities, priorities for archiving data, and future production processes and capabilities.

Increased quality in code development activities: Rigorous and systematic V&V processes lead to more efficient, more accurate code development. In particular, debugging, algorithm assessment, and physics assessment become more systematic and effective. Code development resources can be better leveraged, and productivity of the code development teams increase.

More accurate codes: Codes that result from systematic V&V processes are more robust, have fewer bugs, have better understood accuracy, have more clearly understood weaknesses and limitations, and have clearer future development paths.

Improved user confidence: Higher quality codes have more predictive application to stockpile problems of interest. By strongly linking stockpile requirements to V&V processes, the progress and product of stages of code development are also strongly linked to higher confidence in stockpile assessment and certification.

Requirements Drive Validation and Verification

Historically, design and engineering codes were used as tools to support new weapons development and nuclear testing. These previous computational simulation and modeling capabilities for nuclear weapons were empirically based and due to computer hardware limitations, most weapon assessments were restricted to two-dimensional or crudely zoned three-dimensional models.

With the cessation of nuclear testing and no requirements for new weapon production, the role of computational simulation and modeling has changed. As the stockpile ages, changes in weapon components and materials will require improved physical models and simulations. Efforts to extend the lifetime of the stockpile rely heavily on our ability to perform predictive, integrated, three dimensional computer simulations to assess the stockpile. Predictive three dimensional simulation codes will require improved physics, materials, and engineering models that must be validated by experienced weapon scientists and engineers.

The required physics, materials, and engineering models needed for the science-based stockpile stewardship approach are also described in the Green Book. Physics and theoretical models that reduce empirical approximations will need experiments specifically designed to validate these models, as well as calibrate key parameters used in their implementation. Measurements made in the enhanced surveillance program will be required to validate materials aging models needed to predict the lifetime of weapons components. Modeling of manufacturing processes and techniques will require data from prototype capability to determine which manufacturing techniques reproduce the functional operation of weapons components. Past nuclear and non-nuclear test data that include test failures and mysteries, tests which have large sensitivity to physics and materials models, and tests that had different weapon component manufacturing processes and techniques will provide valuable validation data to establish the highest confidence in future weapons simulations

The V&V teams will be composed of code developers, software engineers, analysts, theoreticians, experimentalists and SSP customers, as well as experienced users of weapons simulation codes, to insure that effective and relevant V&V is carried out. Meeting stockpile assessment and certification requirements will naturally lead to code user acceptance and confidence and is an important measure of success for this program.

This V&V program derives its requirements from existing Green Book stockpile drivers to help us identify the specific validation tasks required to establish confidence in our computational capabilities for stockpile assessment and certification. We expect that some data that will be required will be of a different form and fidelity than we have historically required. Experimental validation data needs will include:

High-fidelity measurements of material physical and chemical properties that are used as inputs to simulation codes
High-fidelity experiments for validation of code submodels/models and coupled phenomena through laboratory and facility tests
Integrated code validation, where possible, through comparison against past nuclear test data, full system tests and stockpile surveillance data

The stockpile assessment and certification deliverables described in the Green Book define part of the requirements that V&V will support. The requirements for the certification of weapons modifications, recertifications, and stockpile refurbishments define capabilities needed for simulation and modeling and a time frame when these capabilities are needed. The specific linkages to Green Book schedules and milestones and ASCI applications development milestones will be updated each year in the V&V Implementation Plan. The needs for new validation data will be documented and communicated to the SSP scientific and integration program through the V&V Experimental Needs document. These documents are described in more detail in the Management Plan.

Program Plan

The validation and verification program has efforts in three areas: improving current verification processes, improving current data comparison techniques for the validation process, and developing a systematic integrated approach for validation and verification of predictive SSP simulations. This program has a number of challenges to address:

Verification of software code with new, rapidly evolving programming models
Verification of software code on new, rapidly evolving computer hardware architectures
Validation of computer simulations with unprecedented levels of spatial, temporal, and spectral resolutions
Validation of computer simulations with unprecedented degrees of multi-physics interactions

To address these new challenges, this program will also examine the development of advanced, innovative approaches to V&V that may be unique to the Stockpile Stewardship Program's need for predictive simulations.

Verification

Rigorous, systematic and comprehensive verification techniques are necessary for designers and analysts to have confidence that simulations represent the conceptual models. Under the SSP, the increasingly complex codes must be created to exacting standards, and there must be means to demonstrate that these codes do in fact conform to the conceptual physical models. Verification, therefore, must address issues of software engineering and comparison to analytic and benchmark numerical results. As illustrated in the bottom portion of Figure 1, the code verification process is performed in parallel with code development. The verification activities in this program will integrate with the application-specific verification activities that are already underway as part of ASCI applications development and problem solving environment (PSE) efforts, as well as legacy code maintenance.

Evaluation of Current Standards

There is a need to re-evaluate common software engineering practices for application to the large-scale software systems that we are building for the SSP under ASCI. The V&V program should involve an assessment of software development standards and their applicability to the SSP. Verification issues to review include:

Are current "standards" and formalisms directly applicable to ASCI software systems? Are they applicable with some modification? Are completely new approaches required?
Are the SEI software maturity model levels appropriate for the software we are building? If so, is there a systematic, logical basis for determining what level is appropriate for our needs? Are there other software engineering standards (e.g., ISO, IEEE) that we should consider?

Development of Methodology and Metrics

Current approaches to software development require extensive computer cycles for the verification of SSP simulations. But since the ASCI platforms are largely capability, rather than capacity resources, one area for advanced development is to examine new ways to verify software modules at a small scale that apply to the large scale. This area is also coupled to the ASCI PSE area of scalable software development tools and code reuse.

Other examples of advanced verification methods that should be explored include:

New metrics for assessing the dynamic quality of integrated software. Such tools might include extensions of current static and dynamic software fault analysis techniques, as well as extensions of the metrics that link such faults to software "correctness."
Development of new bottom-up module verification techniques. This would address issues of not only the best ways to assess the "correctness" of individual modules, but also how to determine the correctness of sets of these modules when integrated. New techniques would include specific concerns about parallel implementations.
Research into the general problem of verifying system software and application code software on large, complex homogeneous or heterogeneous massively parallel computing systems.
Verification Test Suites

Code verification must include detailed comparisons with analytic and benchmark numerical results for one, two and three dimensions. For example, a rigorous test suite can be designed to test the new codes in their ability to resolve the following radiation hydrodynamic phenomena; shock wave interactions, radiation propagation in a moving fluid, track the development of vortex induced flows, track shear flow layers, preserve symmetry properties, account for both radiation induced and thermally induced flows, resolve various hydrodynamic instabilities and calculate thermonuclear burn. We briefly describe an analytic and benchmark solution set of test problems that are designed to verify the radiation hydrodynamic properties of codes. Similar sets of test problems must be designed for other relevant physics. This set of problems is not exclusive and is illustrative only.

Specific comparisons to the performance of current trusted design codes on these problems
Self-similar strong blast waves in uniform and non-uniform media for different geometries
Strongly interacting blast waves
Self-similar strong shock implosions
Self-similar expansion into a vacuum
Strong shock tube propagation with multi-fluid interfaces

simulation a comparison graph

Figure 2. Comparison of analytic solution to RAGE simulation

Analytic Test Problem and Code Comparisons

While this activity is performed under the standard process of application/code development, within the context of this V&V program there is a role to study formal protocols required to perform code comparisons (including ongoing DOE activities such as JOWOG-42), most usefully and successfully.

Validation

Validation requires comparison with observed data and thus is dependent upon comparisons with experimental, archival, or surveillance data. The goal of the validation process is to establish with increasing confidence that a code is predictive. One metric of success for the validation process would be increased confidence in using the code to extrapolate beyond the domain of the experimental data it was validated with.

Historically, experiments were designed to work in parallel with two-dimensional design codes to certify weapons components and systems. This test-based approach provided certification of point designs. The expectation was that modifications to designs would be re-certified with additional tests.

Within the new simulation-based SSP environment, a primary driver for the experimental program is to provide validation data that will be used to establish confidence in simulation capabilities which, in turn, will be used to certify weapons. This confidence will be established by exercising the feedback loop between simulations and experiments as shown in the upper left-hand portion of Figure 1. Experiments must not only assist in the certification of components and systems, they now must be designed to provide the necessary information to establish confidence in predictive simulation codes over a broad parameter space. Conversely, new experiments will also serve to establish regions in physical parameter space where confidence in simulation results is degraded. This new requirement calls for increased fidelity in the collection of experimental data to exercise the regimes of physics that our more powerful and predictive 3D codes can provide.

Requirements for Validation Data

The comparison and analysis of a SSP simulation with a nuclear event or weapon stockpile-to-target-sequence (STS) test requires access to a large amount of data. This includes not just the results of the test diagnostics, but detailed information on all of the device parts, details of the diagnostic system, and data on non-nuclear calibration experiments. As illustrated in the middle of Figure 1, all of this data needs validation.

A necessary first step for validation is that the data against which the codes are validated can be located in a usable form. Unfortunately, much of the historical data has been kept by different organizations by individuals who have left the laboratories. Currently this nuclear weapon data is being gathered by various archiving projects within the DOE complex, but there is much more data yet to be gathered than what is already in the database. V&V data needs should provide guidance and constraints to archiving projects on which data are most important for validation.

In general, evaluations of these validation databases should:

Ensure that all relevant experimental data are included
Identify critical gaps in the experimental data that adversely impact our ability to validate the latest theory and models
Provide a quantitative estimate of the uncertainty in the data and limits of its applicability
Document how each of the evaluations was carried out and how estimates of the uncertainties were established
Make this information available to designers and analysts electronically

When these tools and information are made available to users, assessment of the particular data requirements for resolution of code issues is possible, and the sensitivity of code output to various data parameters can be established. Evaluation of code output sensitivities will naturally lead to higher fidelity experimental needs for appropriate data. These needs must then be communicated from the validation groups to the experimentalists so that joint computational/experimental validation can be conducted.

Simulation and computing capabilities are pushing into new regimes in terms of time and length scales. Because of these new simulation capabilities, experiments and associated diagnostics are needed with the resolution to capture data in these new regimes. The new codes will be required to simulate above ground experiments (AGEX) that have well characterized physics and initial conditions so that realistic code comparisons can be made. This requires that new experiments be designed that employ databases (e.g. opacity, EOS, etc.) with quantified uncertainties. Facilities such as Nova and NIF will allow us to validate a full ASCI code system in high energy density environments and other facilities such as PBFA-Z and ATLAS will allow code validation in regimes relevant to turbulent multiphase flow.

Validation of physical database information usually involves testing at different energy levels. For example, in the case of materials equation of state, the most basic level of validation involves direct static or dynamic high-pressure equation of state experiments on stockpile materials. In the low energy density regime, shock compression, static high-pressure diamond anvil cell, and isobaric expansion experiments have traditionally provided both a large quantity and a wide range of relevant data. In the high energy density regime, strong shock waves are usually needed to generate the required high temperatures. Limited data is available from previous underground experiments and future data collection and equation-of-state validation will rely primarily on experimental campaigns based on the utilization of high-intensity lasers.

nuclear tests document

Figure 3. Unique archive of nuclear test data

Validation Against Data

There are several sources of data for validation of SSP simulations. They will be discussed below as archived data from nuclear tests, archived data from non-nuclear tests, fundamental physics experimental data, system certification tests, and stockpile surveillance data. Obviously data from past experiments and certification tests could be considered part of the archive of non-nuclear tests. For purposes of this program plan, we consider these categories to focus on new validation data that this program could obtain by providing guidance and recommendations to the relevant experimental and archiving programs for future data collection opportunities.

Archived Data from Nuclear Tests

The archive of data from nuclear tests is a unique resource for validation of codes to simulate nuclear weapons performance and safety, and components and system reliability in hostile nuclear environments. This body of data provides an important link to the prior test-based certification process, and thus is a key to assuring confidence in the transition to a simulation-based certification process. Work on V&V should generate useful input to nuclear weapons archiving projects on what data are most important for validation.

Types of nuclear weapon test data relevant to performance and safety cover a broad range from total yield inferred from weapons effects to sophisticated time-resolved diagnostic measurements of device operation. The ensemble of tests includes some where design characteristics were varied systematically and some that were intended to address specific weapon physics or engineering issues. The results of some tests were hard to understand and could be considered challenging mysteries or anomalies. As in any experimental field, the quality of the data varies substantially. More recent tests generally had more sophisticated diagnostics and better quality data.

Stockpile milestones and objectives underlie the code development projects and their associated V&V plans. Issues involving specific weapons systems will require advances in particular physics and engineering models and their numerical implementation. Validation by comparison with nuclear test data will thus be driven by the same stockpile stewardship priorities. Three examples follow that illustrate the role of nuclear test data in validation.

A nuclear performance assessment that depends on prediction of the performance of high explosive under unusual environmental conditions may require development of a higher fidelity explosive detonation model. While this model can be partially validated with non-nuclear experiments, validation of the model as it affects primary performance must ultimately be done by comparing with nuclear test data. In this case, there may not be tests of the system in question under the conditions of interest. Validation would then involve data from tests of similar systems. The most relevant tests will be those that had specialized diagnostics or that were particularly sensitive to the processes that are affected by explosive performance.

prep for underground tests

Figure 4. Preparations for underground nuclear test at NTS

A nuclear performance assessment that depends on the performance of the secondary at full yield may require higher fidelity methods for radiation transport and material response under extreme conditions. Radiation transport and material properties may be partially validated by comparison with non-nuclear experiments at lower temperature, pressure, or on a different time scale. Validation under weapon conditions requires validation against nuclear test data. The most relevant tests will again be ones with specialized diagnostics that were particularly sensitive to the processes that control secondary performance. Validation may require comparing simulations with tests of the system in question and tests of other systems with somewhat different phenomenology.

A nuclear safety assessment may require development of higher fidelity models for primary criticality and yield following abnormal initiation of the high explosive under hypothetical accident conditions. Explosive initiation models can be validated with non-nuclear experiments. However, the consequences for nuclear safety of altered explosive performance after abnormal initiation must be validated by comparison to data from relevant nuclear tests. No single nuclear test is likely to match precisely the postulated initiation conditions. Validation will thus be done by comparison with data from whatever tests there may have been where the explosive was initiated abnormally. An example would be a "one-point safety" test, where the explosive was intentionally initiated at a single point.

Simulation of component and system reliability in hostile nuclear environments can be partially validated against data from non-nuclear experiments involving specialized pulsed radiation facilities. Confidence from validation against data taken during nuclear tests would be much higher because the nuclear environment much better matches conditions of interest. Even data from nuclear tests have limitations because of geometric constraints, treaty constraints on yield and insufficient system characterization, but nuclear test data are a much more comprehensive test of the simulations because the environment has radiation in energy ranges and durations that can only be approximated in non-nuclear tests.

Validation against existing nuclear test data is expected to be a critical part of the V&V plan for ASCI nuclear performance and hostile environments code development projects. The overall project plans will have been developed to address stockpile assessment and certification issues and deliverables. This sets the priorities and time phasing for new or improved physics, materials, and engineering theoretical methods and models, and their implementation in numerical algorithms. As in the examples given here, the code plans and stockpile drivers define the types of data and nuclear tests that are most relevant. The code V&V plans will outline how the nuclear test data will be used for validation.

flight test

Figure 5. Preparations for JTA flight test

An important element of validation is robustness to develop confidence in predictions for a range of problems, not just for the validation test suite. This is particularly needed for validation against nuclear test data because systems of interest are unlikely to match a specific test configuration. Confidence in robustness can be gained by using the same code to perform calculations and compare results with data from a number of nuclear tests.

Archived Data from Non-Nuclear Tests

Assessments of nuclear performance of devices depend not only on evaluation against archival nuclear data, but three other types of archival non-nuclear data as well. During development of a weapon system, a series of device hydrodynamics experiments were performed to characterize the implosion dynamics of a device. Another series of engineering weaponization tests characterized the full-system response to storage, transportation, and STS requirements. And finally, the physics underlying the code implementations relies on a large database of physical data characterizing the material behaviour in the weapon.

As with archival nuclear data, design and diagnostic characteristics of device hydrodynamic tests varied. The quality of each data set varied considerably in terms of many parameters including relevance to War Reserve (WR) materials and geometry, completeness of diagnostic suite applied, quantity and uncertainty of the data returned. In general, the geometry of devices tended to evolve throughout the test sequence so that only the latest hydro tests were close to WR configurations. Types of hydro tests ranged from nominal performance tests in surrogate and stockpile materials to non-nominal performance tests with geometries varying from nominal. As better diagnostics and requirements for improved 2D and 3D code validation data have evolved, increasing reliance has been placed upon more modern and different types of hydro data. However, currently the archival hydro data is still the major source of hydro data for many weapon systems.

A second large body of archival data is the development phase engineering and weaponization test data which characterized the response and survivability of components and the full weapon system to storage, transportation, and Normal, Hostile, and Abnormal STS requirements.

As with data from nuclear tests, the engineering simulations can be partially validated against small-scale and component tests. However, demonstrating confidence in engineering system simulations to address stockpile stewardship issues requires validation against large-scale system tests. Examples are Joint Test Assembly (JTA) flight tests, hostile x-ray impact tests, and large-scale target penetration tests. Examples of data from these tests for validation of simulations of the nuclear explosive package include acceleration, stress, strain, and damage histories and measurements of delivered system function. Non-nuclear component test examples include Arming, Fuzing, and Firing response to hostile x-ray or abnormal fire environments, and Re-entry Vehicle response to deployed vibration and thermal conditions. The archiving of these types of data has not progressed even as far as the archiving for the nuclear and hydro data. The archiving of this non-nuclear data will be required before the loss of this weaponization corporate memory is irreversible.

penetration test

Figure 6. B61/11 target penetration test

A third large body of archival data is the physical properties database underlying all of the nuclear and non-nuclear modeling. The physical material property data (EOS, opacity, cross sections, constitutive data) also requires assessment based upon the requirements for validation stated above. The codes used to evalute and generate these physical databases will also require validation just as the hydro, nuclear and engineering/weaponization codes.

Databases are generally independent of specific hydro, nuclear or engineering codes and are accessed through auxiliary code routines that read the data files and present requested data to the code. Depending on the data set and physical state represented, accuracy varies from precise match to good experimental data to "bad enough to crash the codes." There is little or no help from the databases or access routines to tell a user anything about the accuracy of the generated data, and where auxiliary documentation does provide this information, it usually does not reside online in a central location, easily accessible to users.

Fundamental Physics Experimental Data

New experiments will be required to test the new ASCI codes in three-dimensional regimes for which previous data is not appropriately detailed, or does not exist. These include shear flow layers, multiply shocked flows, vorticity producing regions and regions that are dominated by Richtmyer Meshkov, Rayleigh Taylor, Kelvin Helmholtz and bending mode instabilities. Some examples of new types of experiments that are being fielded to obtain data in these regimes are:

Shock-sphere interactions including Rayleigh Taylor, Kelvin Helmholtz and Richtmyer Meshkov instabilities
Hydrodynamic jet propagation
Single and multi-mode perturbation growth from shock ablation-driven interfaces
Spherical mixing layers to capture the growth of instabilities on non-steady shock accelerated spherical and cylindrical interfaces

Current experimental capabilities have evolved to include more extensive or more accurate diagnostics on traditional types of experiments to the development of new kinds of experiments generating completely new types of data. For example, traditional diagnostics on certain hydro tests have been extended to cover a more complete range of the performance regime of devices and new diagnostics have been added to measure specific physical phenomena with greater accuracy. Completely new kinds of experiments have extended the utility of hydrodynamics experiments in characterizing device behaviour (in both 2D and 3D) up until the generation of nuclear energy. New types of contained subcritical experiments in stockpile materials will elucidate hydro properties not measured previously.

Coupled computational/experimental activities will play an important role in the design and optimization of these high-fidelity experimental programs to ensure that experimental conditions and instrumentation are appropriate for the generation of the data that is required for code validation. Examples of laboratory-scale experiments include those that capture spatially, temporally, and spectrally resolved data to compare with multi-dimensional simulation results. Simulation results can also be used to design lab-scale experiments to capture data around boundary layers, critical points, and other loci of relevant physics.

Laboratory scale above ground experimental (AGEX) experiments at new kinds of facilities such as gas gun, high energy laser, and pulsed-power experiments provide new types of data in physical regimes relevant to nuclear weapons, and provide opportunities for understanding physical phenomena in 3D and dimensionally fine detail not available during weapon development.

B1 dropping B61/11

Figure 7. B61/11 certification flight test

System Certification Test Data

This program will provide the ability to identify and follow through on opportunities for adding diagnostics and instrumentation to full system certification tests that would help capture unique data that might otherwise be lost for validation purposes. With the increased complexity of the required experiments, due to the need for higher fidelity data, and restrictions on the number of laboratory and full system tests, the role of simulation in the design of experiments is more important. Simulation results can be applied to help set data acquisition requirements by providing experimenters with diagnostics requirements: instrumentation dynamic range, sensor locations/orientations, data rates, etc.

Evolution in both diagnostics and new types of tests provide new capabilities for engineering and weaponization characterizations. This testing is of particular importance in assessing the condition of aged materials, components, and systems in the stockpile. In particular, modeling of aging in materials has been limited at least partly by a lack of aging data. In addition, many STS requirements have been modified since the weapon systems’ deployment in the stockpile, and new testing must be performed to revalidate the systems to their new requirements.

Stockpile Surveillance Data

There are very few sources of validation data for materials aging simulations. The ASCI materials aging application development teams have ongoing interactions with the Enhanced Surveillance Program. This program will build on these interactions to obtain the data that we need to validate our aging simulations. These aging simulations may point out problems to look for, and may even suggest measurement techniques and ranges for future surveillance activities.

surveillance photo

Figure 8. Surveillance finding: bridge wire corrosion

Advanced Validation Tools

High consequence computing is one of the implications of simulation-based stockpile stewardship. Because of this key role, the V&V program needs to move aggressively from the past paradigm of code calibration, i.e., adjustment of modeling parameters to attain agreement with experimental data, to a more rigorous paradigm of validation science.

Validation science, which relies upon the development and deployment of a host of advanced tools and methodologies, will aim to accomplish the following objectives:

Encourage the application of techniques for improved computationally based, quantitative experimental data - calculated data comparisons. The same goal can be achieved for code-to-code comparisons using these tools.
Achieve improved and quantitative understanding of the selection of data most appropriate for the code validation task.
Determine more precise and quantitative implications of accurate or inaccurate computational comparison with the selected experimental data for the code validation task.
Minimize the impact of human error on the computational component of the SSP through application of the code validation process.

We expect that a wide range of tools and approaches already exist that can be applied to these tasks, including past Stockpile Computing validation approaches. However, ASCI hardware and software environments have unique scope and capability. There is little reason to believe that the resulting code validation challenge will rest completely on existing capabilities and strategies. We anticipate that knowledge and capability in the area of validation science will evolve from presently understood and applied techniques.

One approach that can provide a systematic foundation for validation science is uncertainty quantification (UQ). This is the technology for quantitatively estimating the uncertainty in the output of code calculations from characterizations of input uncertainty, including underlying model uncertainties and existing validation database. One of the results of applying UQ is that confidence in the results of code calculations can be quantitatively assessed.

In addition, UQ can serve as an organizing principle for the code-experimental data comparison activity. For example, UQ provides intrinsic data about parts of a calculation that contain the greatest degree of uncertainty. It is then logical to drive the validation activity, for that particular application, with those experimental data that are most relevant to that uncertainty contribution. If these data do not exist, this also provides a rigorous rationale for guiding new experimental activity. UQ also suggests rigorous strategies for extrapolating code confidence from specific, successful data comparisons. We anticipate that additional work will have to be performed to make UQ as effective as possible in the SSP code validation activity.

Integrated Validation and Verification

While it is important to establish and improve specific validation and verification techniques, this program must also ensure that a well-integrated validation and verification process is in place. This involves issues such as planning, process improvement, and independent code comparisons and reviews. Many of the validation and verification activities described above are already underway as part of the individual code development team plans. The integrated activities described below are beyond the scope of any single code development project.

Effective Validation and Verification Planning

Because V&V is so crucial to the development of high quality, predictive SSP codes, it is important for each code project to have a well-documented V&V process appropriate to the software development task. This process should address V&V activities that occur at all stages of software development. The accompanying V&V plan should at a minimum set out a clear set of V&V goals, explain the various techniques and tools to accomplish V&V (especially for verification), and describe the data from joint computational/experimental activities needed for code validation. A goal of this V&V program is to ensure that all code projects have appropriately developed V&V plans.

Process Improvement Tools and Techniques

This V&V Program will provide the resources necessary to investigate, assess, and apply the "best practices" and standards of industry and academia as appropriate for large-scale scientific simulations. As described above in the section on Verification, evaluation of standards and development of methodology and metrics are important elements of the verification process. In general, these verification activities need to be performed at the high level of this V&V Program. The outcome of this activity should be a determination of tools, techniques, benchmarking practices and methodologies that are generally applicable to the verification of ASCI-scale scientific simulations. Each code development team will be responsible for using and integrating these high level standards and methodologies for their application.

For validation, process improvement means that this program will seek to define protocols for code usage, specify techniques for comparing simulation results to validation data along with their associated uncertainties and errors, and specify validation data requirements to measure the predictive nature of the codes. A key role for the V&V Program will be to collect, organize, and communicate our needs for validation data to the organizations and programs that own the validation data.

Independent Technical Reviews

While it is expected that the code projects will incorporate V&V, an additional measure of independence is required. The V&V program will establish a program-wide software technical review process that introduces the necessary independence through, for example, cross-lab peer review or external review. Procedures and protocols will be modeled after the JOWOG-42 activities. At some future point in time, this V&V activity could be factored into the ongoing JOWOG program.

An important step in V&V will be detailed comparisons between various codes to ensure that the new ASCI level codes can faithfully reproduce the results of previous legacy codes in their domain of validity or trust. This element of trust is the trust of the designers and analysts that have experience in the application of legacy codes to perform stockpile assessments and certifications.

Challenge Problems

Ultimately, the development of confidence in the predictive capability of SSP simulations will rely on more than just the systematic, rigorous validation and verification of codes and underlying physical models. This V&V program will also work with the experimental elements of the SSP to establish structured challenge problems that provide an opportunity for published (within the stockpile stewardship community) pretest predictions.

A carefully designed challenge problem component will be significant to completing the shift from test-based to simulation-based certification. The use of simulations to make pretest predictions that can subsequently be validated or refuted with new experimental test results involves risk. In fact, the expectation is that challenge problems will lead to instances of spectacular simulation failures, but these failures are analogous to test failures at the NTS. Often, we learn more from the failures than the successes. The eventual outcome of simulation failures will be improvements in the simulations and ultimately, increased confidence in both the validated domain of a simulation capability and increased understanding of its limits. Note that if challenge problems never lead to simulation failures, we need to go back and find more difficult problems. It is by overcoming challenge problems that we will gain demonstrable confidence in our simulations.

Management Plan

Personnel

Executive Committee

A V&V executive committee will be responsible for the day-to-day management of this program. This committee should include two high-level representatives (a primary and an alternate) from each laboratory and from Defense Programs headquarters. It is important that the committee be formed so that it has a comprehensive view of validation and verification. This committee should hold quarterly face-to-face meetings and should review the entire program annually.

Validation and Verification Teams

The most constructive validation environment will require the formation of V&V teams consisting of computational and experimental experts who will define the code validation requirements and the experiments required to meet those requirements. As noted above, validation requirements are set by the application. To obtain the perspective of the application, designers and analysts also need to be part of this team. These experts should be chosen from the current ASCI program (perhaps by the laboratory ASCI application managers) and existing experimental and certification programs.

Management Objectives

Ensure proper focus and emphasis of Validation and Verification to improve the quality of codes
Facilitate interactions between laboratories
Establish a strong linkage between V&V requirements and other SSP needs
Encourage development of standards for Validation and Verification of codes and databases
Continue to promote investigations into improved techniques and approaches

Management Processes and Tools

Validation and Verification Program Plan. This program plan provides a description of the overall direction and policy for V&V activities. This document is a primary vehicle for fostering interest in, and understanding of the need for, validation data in the portions of the SSP that have responsibility for generating or collecting this validation data. More detailed information on specific activities and needs will be available in the V&V Implementation Plan and Experimental Needs documents.
Validation and Verification Implementation Plan. A coordinated Tri-lab implementation plan will be used as a tool for yearly planning and to describe the projects that are funded in the V&V program.
Validation and Verification Experimental Needs Document. To coordinate between ASCI applications and the experimental program, the V&V program will be responsible for establishing a comprehensive set of experimental needs and communicating those needs to experimentalists and management across laboratories and program offices under the Assistant Secretary for Defense Programs. This document is separate from the V&V program plan and will be developed jointly by representatives from the experimental program, designers and analysts, and code developers. Periodically, as needs are satisfied and new issues arise, the document will be updated.
External Review. An external management review process will be used to assess whether the Validation and Verification program resources are being used cost effectively and program goals are being met. This will be different from the independent technical review process described earlier.

Relationship to Other Stockpile Stewardship Program Elements

Accomplishing an effective V&V program requires coordination with several other programs. This is due to the fact that:

Much of the validation data is owned, managed, or collected outside of code development.
Designers will establish their confidence in codes through understanding and participating in the V&V process.
Each ASCI application development team has a V&V effort underway.
Validation and Verification provides critical input into the certification and acceptance process.

The SSP elements we will coordinate with are:

Surveillance, Manufacturing & Operations
Scientific & Experimental Integration
Strategic Computing & Simulation
Surveillance, Manufacturing and Operations

Ongoing weapon certification and revalidation requirements will rely increasingly on simulations as weapons enter regimes where archival test data and legacy design codes become less relevant. As the end users of ASCI codes, designers and analysts must lead the validation of codes developed in order to have confidence when they make judgments based on the codes' results. The design community will be involved in the entire spectrum of validation activities. Verification work is also necessary to prove to designers and analysts that coding and configuration errors have been minimized.

The surveillance data necessary for validation of materials aging simulations will be collected by this element of the stockpile stewardship program. These simulations will then be used to predict the effects of aging on weapons reliability, safety and performance. The validation of new manufacturing process simulations to support rapid prototyping will also require coordination with this program element. This element is also responsible for integrated flight tests and full system certification tests of the stockpile. While the number of these tests may be reduced due to budgetary constraints, that fact makes them even more valuable as potential opportunities for collection of STS validation data.

Scientific and Experimental Integration

Experimental Programs

The experimental resources (facilities, expertise, etc.) in the core R&D program are essential for validation activities that compare simulation results to physical data. The validation program will become more tightly coupled to the experimental program through a variety of activities:

New experiments will be designed with instrumentation specifically needed for comparison with physical models and aggregate effects in codes.
Existing experimental activities will be redesigned so that the data can be leveraged for validation.
Experimentalists will benefit from simulation capabilities that can evaluate different experimental approaches and increase the benefit of the limited experimental resources.

This coupling will exist throughout the experimental program from the small laboratory experiments to the large multi-million dollar experimental facilities.

Archiving Programs

The Stockpile Stewardship Program also owns the archive of nuclear and non-nuclear test data and has underway the programs that are responsible for the archiving, organizing and preserving past test results. This V&V program provides an opportunity to systematically review and prioritize our requirements for validation data from the test archives. This program has the responsibility to organize and communicate the validation needs of all SSP simulations to the archiving programs with a common voice.

Strategic Computing and Simulation

ASCI Advanced Applications Development

Validation and Verification is an integral part of the development of ASCI applications codes. Each code development team has specific approaches to V&V for their codes. Each code will benefit from the management of an integrated V&V program that looks across code development efforts to compare V&V methods and requirements. The code development teams have some common needs for tools to perform V&V tasks and to establish methodologies for validation activities such as comparison to experimental images. The coordination and resources provided by the V&V program will be used to identify and fill these common requirements that are needed by several code teams or that fall between responsibilities of code development, designers, PSE, or the experimental program.

ASCI Problem Solving Environment

One of the strategies of the PSE Advanced Software Environment (ASE) effort is to address issues that improve the reliability of large parallel modeling and simulation codes. To address this strategy ASE provides computer and computational science support to the various ASCI application code teams and is responsible for the specification and deployment of common software development tools. The ASE project also provides direct support for validation and verification with an emphasis on code verification, software quality assurance and operating system/compiler reliability through the development of tools for the design and analysis of computer experiments. In addition, ASE provides common tools to facilitate the rapid development of quality high-performance application codes.

These efforts are already underway and share many of the same long-term goals of the V&V efforts. The V&V program will coordinate its activities with the ongoing activities of PSE and leverage the PSE efforts wherever possible to achieve maximum effect.

ASCI Alliances and Collaborations

One of the goals of the ASCI Academic Strategic Alliance Program (ASAP) is to establish and validate the practices of large scale modeling, simulation and computation as a viable scientific methodology. Since validation and verification are important practices in large scale computer simulation, it is expected that the universities involved in ASAP will have much to contribute to improving approaches to V&V; for example, university centers that are developing large scale computer codes as part of ASAP should be expected to produce V&V plans and strategies in much the same manner as the ASCI code projects. This is especially true since the ASAP program is expected to establish technical coupling of Strategic Alliances efforts with ongoing ASCI projects in DOE laboratories.

Another important goal of ASAP is to accelerate advances in critical computer science areas, in computational science and engineering, in high performance computing systems and in problem solving environments that support long-term ASCI needs. Ensuring that code projects apply the appropriate means to achieve high confidence in simulations is an extremely important ASCI need, and it is expected that research results from ASAP will apply to improving V&V methodologies and techniques.

Stockpile Computing

The Stockpile Computing program is responsible for the maintenance of our legacy codes. The coordination of the V&V program with Stockpile Computing addresses two issues. A portion of the V&V process will use comparisons of new ASCI simulation results with results from the legacy codes that our weapons designers and analysts have some established confidence in. The validated and verified ASCI codes will eventually become the production codes of the future as such they will eventually transition to the custody of Stockpile Computing. This transition will occur when simulation-based weapons assessment and design is part of the formal certification and acceptance process for the SSP.

NEWS

The NEWS (Numerical Environment for Weapons Simulation) program will start in parallel with this V&V program. The focus for NEWS is on establishing the infrastructure for visualization and understanding of the results from ASCI simulations. The requirements for validation and verification of these simulations will undoubtedly lead to the need to be able to compare these ASCI simulation results to experimental, archival and surveillance data. This program will work to coordinate these needs with the NEWS program.

DisCom²

The DisCom² (Distance and Distributed Computing) program will also start at the same time as this V&V program. The distance strategy portion of this program will provide the mechanism for connecting the laboratories to the plants. This aspect of the DOE defense program's distributed enterprise will provide a unique opportunity for validation of manufacturing process simulations by supporting the collection of manufacturing data from the plants. This system will also provide an opportunity to support the comparison of simulation results with archival test results by providing on-line linkages to "as built" data at the plants for the components and systems that were tested. Again, this program will coordinate these validation needs with DisCom².

Acronyms

AGEX

Above Ground Experiment

ASAP

Academic Strategic Alliances Program

ASCI

Accelerated Strategic Computing Initiative

ASE

Advanced Software Environment

DisCom²

Distance and Distributed Computing

DOD

Department of Defense

EOS

Equation of State

IEEE

Institute of Electrical and Electronics Engineers

ISO

International Standards Organization

JOWOG

Joint Working Group (US-British)

JTA

Joint Test Assembly

NEWS

Numerical Environment for Weapons Simulation

NTS

Nevada Test Site

PSE

Problem Solving Environment

SEI

Software Engineering Institute

SSP

Stockpile Stewardship Program

STS

Stockpile-to-Target-Sequence

Uncertainty Quantification

V&V

Validation and Verication

War Reserve