Publications

Abstract not provided.

This report describes the underlying principles and goals of the Sandia ASCI Verification and Validation Program Validation Metrics Project. It also gives a technical description of two case studies, one in structural dynamics and the other in thermomechanics, that serve to focus the technical work of the project in Fiscal Year 2001.

It is critically important, for the sake of credible computational predictions, that model-validation experiments be designed, conducted, and analyzed in ways that provide for measuring predictive capability. I first develop a conceptual framework for designing and conducting a suite of physical experiments and calculations (ranging from phenomenological to integral levels), then analyzing the results first to (statistically) measure predictive capability in the experimental situations then to provide a basis for inferring the uncertainty of a computational-model prediction of system or component performance in an application environment or configuration that cannot or will not be tested. Several attendant issues are discussed in general, then illustrated via a simple linear model and a shock physics example. The primary messages I wish to convey are: (1) The only way to measure predictive capability is via suites of experiments and corresponding computations in testable environments and configurations; (2) Any measurement of predictive capability is a function of experimental data and hence is statistical in nature; (3) A critical inferential link is required to connect observed prediction errors in experimental contexts to bounds on prediction errors in untested applications. Such a connection may require extrapolating both the computational model and the observed extra-model variability (the prediction errors: nature minus model); (4) Model validation is not binary. Passing a validation test does not mean that the model can be used as a surrogate for nature; (5) Model validation experiments should be designed and conducted in ways that permit a realistic estimate of prediction errors, or extra-model variability, in application environments; (6) Code uncertainty-propagation analyses do not (and cannot) characterize prediction error (nature vs. computational prediction); (7) There are trade-offs between model complexity and the ability to measure a computer model's predictive capability that need to be addressed in any particular application; and (8) Adequate quantification of predictive capability, even in greatly simplified situations, can require a substantial number of model-validation experiments.

The thermal conductivity of 304 stainless steel has been estimated from transient temperature measurements and knowing the volumetric heat capacity. Sensitivity coefficients were used to guide the design of this experiment as well as to estimate the confidence interval in the estimated thermal conductivity. The uncertainty on the temperature measurements was estimated by several means, and its impact on the estimated conductivity is discussed. The estimated thermal conductivity of 304 stainless steel is consistent with results from other sources.

Computational models have the potential of being used to make credible predictions in place of physical testing in many contexts, but success and acceptance require a convincing model validation. In general, model validation is understood to be a comparison of model predictions to experimental results but there appears to be no standard framework for conducting this comparison. This paper gives a statistical framework for the problem of model validation that is quite analogous to calibration, with the basic goal being to design and analyze a set of experiments to obtain information pertaining to the `limits of error' that can be associated with model predictions. Implementation, though, in the context of complex, high-dimensioned models, poses a considerable challenge for the development of appropriate statistical methods and for the interaction of statisticians with model developers and experimentalists. The proposed framework provides a vehicle for communication between modelers, experimentalists, and the analysts and decision-makers who must rely on model predictions.

The objective of agile manufacturing is to provide the ability to quickly realize high-quality, highly-customized, in-demand products at a cost commensurate with mass production. More broadly, agility in manufacturing, or any other endeavor, is defined as change-proficiency; the ability to thrive in an environment of unpredictable change. This report discusses the general direction of the agile manufacturing initiative, including research programs at the National Institute of Standards and Technology (NIST), the Department of Energy, and other government agencies, but focuses on agile manufacturing from a statistical perspective. The role of statistics can be important because agile manufacturing requires the collection and communication of process characterization and capability information, much of which will be data-based. The statistical community should initiate collaborative work in this important area.

Component test plans are often designed by allocating a system's reliability goal among the system's components, then designing individual component test plans suitable for demonstrating achievement of each component's reliability goal. One use of the resulting component test data is the calculation of estimated system reliability, based on a model linking the component reliabilities to system reliability. The statistical precision of this system estimate depends on the component test plans (numbers of each component tested and the type of tests, e.g., variables or attributes) and, hence, is determined by the component test planners. Because system reliability may be of considerable interest, we feel an integrated view of component testing is required to assure that the ensemble of component tests will provide an adequate system reliability estimate. This paper considers the case of a series system of different components and binomial component data. For the case of equal numbers of units tested of each component (which can be shown to minimize total cost, subject to the risk constraints) the O.C. envelope is readily derived and from this envelope component test plans that satisfy the specified risks can be derived from equations that involve the cumulative binomial distribution function. Existing tables pertaining to acceptance sampling plans based on the binomial distribution can be used to determine the required number of component tests. 10 refs., 2 figs.