We introduce a robust verification tool for computational codes, which we call Stochastic Robust Extrapolation based Error Quantification (StREEQ). Unlike the prevalent Grid Convergence Index (GCI) [1] method, our approach is suitable for both stochastic and deterministic computational codes and is generalizable to any number of discretization variables. Building on ideas introduced in the Robust Verification [2] approach, we estimate the converged solution and orders of convergence with uncertainty using multiple fits of a discretization error model. In contrast to Robust Verification, we perform these fits to many bootstrap samples yielding a larger set of predictions with smoother statistics. Here, bootstrap resampling is performed on the lack-of-fit errors for deterministic code responses, and directly on the noisy data set for stochastic responses. This approach lends a degree of robustness to the overall results, capable of yielding precise verification results for sufficiently resolved data sets, and appropriately expanding the uncertainty when the data set does not support a precise result. For stochastic responses, a credibility assessment is also performed to give the analyst an indication of the trustworthiness of the results. This approach is suitable for both code and solution verification, and is particularly useful for solution verification of high-consequence simulations.
The Saturn accelerator has historically lacked the capability to measure time-resolved spectra for its 3-ring bremsstrahlung x-ray source. This project aimed to create a spectrometer called AXIOM to provide this capability. The project had three major development pillars: hardware, simulation, and unfold code. The hardware consists of a ring of 24 detectors around an existing x-ray pinhole camera. The diagnostic was fielded on two shots at Saturn and over 100 shots at the TriMeV accelerator at Idaho Accelerator Center. A new Saturn x-ray environment simulation was created using measured data to validate. This simulation allows for timeresolved spectra computation to compare the experimental results. The AXIOM-Unfold code is a new parametric unfold code using modern global optimizers and uncertainty quantification. The code was written in Python, uses Gitlab version control and issue tracking, and has been developed with long term code support and maintenance in mind.
The AXIOM-Unfold application is a computational code for performing spectral unfolds along with uncertainty quantification of the photon spectrum. While this code was principally designed for spectral unfolds on the Saturn source, it is also relevant to other radiation sources such as Pithon. This code is a component of the AXIOM project which was undertaken in order to measure the time-resolved spectrum of the Saturn source; to support this, the AXIOM-Unfold code is able to process time-dependent dose measurements in order to obtain a time-resolved spectrum. This manual contains a full description of the algorithms used by the method. The code features are fully documented along with several worked examples.
This paper describes the verification and validation (V&V) framework developed for the stochastic Particle-in-Cell, Direct Simulation Monte Carlo code Aleph. An ideal framework for V&V from the viewpoint of the authors is described where a physics problem is defined, and relevant physics models and parameters to the defined problem are assessed and captured in a Phenomena Identification and Ranking Table (PIRT). Furthermore, numerous V&V examples guided by the PIRT for a simple gas discharge are shown to demonstrate the V&V process applied to a real-world simulation tool with the overall goal to demonstrably increase the confidence in the results for the simulation tool and its predictive capability. Although many examples are provided here to demonstrate elements of the framework, the primary goal of this work is to introduce this framework and not to provide a fully complete implementation, which would be a much longer document. Comparisons and contrasts are made to more usual approaches to V&V, and techniques new to the low-temperature plasma community are introduced. Specific challenges relating to the sufficiency of available data (e.g., cross sections), the limits of ad hoc validation approaches, the additional difficulty of utilizing a stochastic simulation tool, and the extreme cost of formal validation are discussed.
This report reviews a hierarchy of formal mathematical models for describing plasma phenomena. Starting with the Boltzmann equation, a sequence of approximations and modeling assumptions can be made that progressively reduce to the equations for magnetohydrodynamics. Understanding the assumptions behind each of these models and their mathematical form is essential to appropriate use of each level of the hierarchy. A sequence of moment models of the Boltzmann equation are presented, then focused into a generalized three-fluid model for neutral species, electrons, and ions. This model is then further reduced to a two-fluid model, for which Braginskii described a useful closure. Further reduction of the two-fluid model yields a Generalized Ohm's Law model, which provides a connection to magnetohydrodynamic approaches. A verification approach based on linear plasma waves is presented alongside the model hierarchy, which is intended as an initial and necessary but not sufficient step for verification of plasma models within this hierarchy.
This report documents recent code verification exercises for a warm electron diode problem using the EMPIRE plasma code. This computationally expensive test was performed three times, including two different code versions and two different time integration algorithms, and the resulting code responses were analyzed for convergence to the analytical solution and orders-of-convergence using the StREEQ numerical error estimation tool. Significant deviations from the exact solution and expected orders-of-convergence, as well as changes in code behavior over time and due to choice of time integration algorithm were observed, illustrating the need to fix fundamental code issues as well as additional code verification testing before the code can be relied upon to accurately solve critical problems within its application space.
Systematic veri cation and validation (V&V) is necessary to establish the credibility for high consequence simulations. In this paper, we focus on a radiation-induced plasma experimental validation exercise for simulations which uses both numerical error estimation and input parameter uncertainty quanti cation to provide a direct comparison between Particle-In-Cell (PIC) plasma simulations and experiments. This approach demonstrates how careful validation can uncover missing physics in the simulation. Three di erent validation examples are shown; a vacuum space charge limited cavity, a gas lled space charge limited cavity, and a vacuum non space charge limited cavity. Two of the example are picked to show the importance of error estimation in uncovering inaccuracy/incomplete simulation models. We also report on a newly-developed numerical error estimation approach, StREEQ, which is a notable improvement to past approaches. In the StREEQ approach, a multi- tting scheme based on L1, L2, and L$\infty$ error norms and alternate weightings is used to propagate uncertainties in the relative importance of outliers and coarse/re ned discretization levels. Bootstrap sampling is used to represent the stochasticity in the response data. The resulting method appears to robustly and conservatively predict the fully-converged response within estimated numerical error bounds for stochastic simulations. The StREEQ approach is demonstrated on two related prototype electron diode problems, and preliminary results are reported for a radiation-induced plasma simulation.
We present a numerical error estimation technique specifically formulated to deal with stochastic code output with multiple discretization parameters. This method is based on multiple fits to an error model with arbitrary convergence rates and cross-coupling terms, performed using nonlinear optimization. The fitting approach varies by the type of residual norm which influences the importance of outliers, and weights which influences the relative importance of data points in the coarse and refined regions of discretization parameter space. To account for the influence of stochastic noise, these fits are performed on multiple bootstrap values based on the underlying response data set. Using an automated discretization domain selection scheme, the fits are performed on a series of reduced sets of discretization levels in order to find an optimal fully-converged result estimate in the minimum variance sense; this automated approach enables straightforward analysis of multiple quantities of interest and/or time and spatially-dependent response data. The overall numerical error analysis method is useful for verification and validation problems for stochastic simulation methods and forms a key component in the overall uncertainty quantification process. The method was demonstrated for steady and unsteady electron diode problems simulated using a particle-in-cell kinetic plasma code, demonstrating excellent results.
The Light Initiated High Explosive (LIHE) facility performs high rigor, high consequence impulse testing for the nuclear weapons (NW) community. To support the facility mission, LIHE's extensive data acquisition system (DAS) is comprised of several discrete components as well as a fully integrated system. Due to the high consequence and high rigor of the testing performed at LIHE, a measurement assurance plan (MAP) was developed in collaboration with NW system customers to meet their data quality needs and to provide assurance of the robustness of the LIHE DAS. While individual components of the DAS have been calibrated by the SNL Primary Standards Laboratory (PSL), the integrated nature of this complex system requires verification of the complete system, from end-to-end. This measurement assurance plan (MAP) report documents the results of verification and validation procedures used to ensure that the data quality meets customer requirements.