ALEGRA is a multiphysics finite-element shock hydrodynamics code, under development at Sandia National Laboratories since 1990. Fully coupled multiphysics capabilities include transient magnetics, magnetohydrodynamics, electromechanics, and radiation transport. Importantly, ALEGRA is used to study hypervelocity impact, pulsed power devices, and radiation effects. The breadth of physics represented in ALEGRA is outlined here, along with simulated results for a selected hypervelocity impact experiment.
Efficient modeling of low magnetic Reynolds number (low-Rm) magnetohydrodynamics is often challenging and requires the implementation of innovative techniques to avoid key barriers experienced with prior approaches. We detail a new paradigm for first-principles simulation of the solution to the low-Rm governing equations in complex geometries. As a result of a number of innovative numerical advances, the next-generation GPU (graphics processing unit) accelerated physics code LGR has been successfully applied to the modeling of exploding wire problems.
As the push towards exascale hardware has increased the diversity of system architectures, performance portability has become a critical aspect for scientific software. We describe the Kokkos Performance Portable Programming Model that allows developers to write single source applications for diverse high performance computing architectures. Kokkos provides key abstractions for both the compute and memory hierarchy of modern hardware. Here, we describe the novel abstractions that have been added to Kokkos recently such as hierarchical parallelism, containers, task graphs, and arbitrary-sized atomic operations. We demonstrate the performance of these new features with reproducible benchmarks on CPUs and GPUs.
The final review for the FY20 Advanced Simulation and Computing (ASC) Integrated Codes (IC) L2 Milestone #7181 was conducted on August 31, 2020 at Sandia National Laboratories in Albuquerque, New Mexico. The review panel unanimously agreed that the milestone has been successfully completed. Roshan Quadros (1543) led the milestone team and various members from the team presented the results. The review panel was comprised of staff from Sandia National Laboratories Albuquerque and California that are involved with computational engineering modeling and analysis. The panel consisted of experts in the fields of solid modeling, discretization, meshing, simulation workflows, and computational analysis including personnel Brett Clark (1543, Chair); Jay Foulk (8363); Jackie Moore (1553); Ron Kensek (1341); Ed Hoffman (8753); Dan Ibanez (1443). The presentation documented the technical approach of the team and summarized the results with sufficient detail to demonstrate both the value and the completion of the milestone. A separate SAND report was also generated with more detail to supplement the presentation. The purpose of the milestone was to advance capabilities for automatically finding, displaying, and resolving geometric overlaps in CAD models.
Using LGR on ATS-2 hardware, we have simulated the performance of nearly ten thousand possible component designs, allowing designers to map and assess the design space prior to experimental testing.
Tetrahedral finite element workflows have the potential to drastically reduce time to solution for computational solid mechanics simulations when compared to traditional hexahedral finite element analogues. A recently developed, higher-order composite tetrahedral element has shown promise in the space of incompressible computational plasticity. Mesh adaptivity has the potential to increase solution accuracy and increase solution robustness. In this work, we demonstrate an initial strategy to perform conformal mesh adaptivity for this higher-order composite tetrahedral element using well-established mesh modification operations for linear tetrahedra. We propose potential extensions to improve this initial strategy in terms of robustness and accuracy.
This report documents the completion of milestone STPRO4-6 Kokkos Support for ASC applications and libraries. The team provided consultation and support for numerous ASC code projects including Sandias SPARC, EMPIRE, Aria, GEMMA, Alexa, Trilinos, LAMMPS and nimbleSM. Over the year more than 350 Kokkos github issues were resolved, with over 220 requiring fixes and enhancements to the code base. Resolving these requests, with many of them issued by ASC code teams, provided applications with the necessary capabilities in Kokkos to be successful.
Simulating energetic materials with complex microstructure is a grand challenge, where until recently, an inherent gap in computational capabilities had existed in modelling grain-scale effects at the microscale. We have enabled a critical capability in modelling the multiscale nature of the energy release and propagation mechanisms in advanced energetic materials by implementing, in the widely used LAMMPS molecular dynamics (MD) package, several novel coarse-graining techniques that also treat chemical reactivity. Our innovative algorithmic developments rooted within the dissipative particle dynamics framework, along with performance optimisations and application of acceleration technologies, have enabled extensions in both the length and time scales far beyond those ever realised by atomistic reactive MD simulations. In this paper, we demonstrate these advances by modelling a shockwave propagating through a microstructured material and comparing performance with the state-of-the-art in atomistic reactive MD techniques. As a result of this work, unparalleled explorations in energetic materials research are now possible.
Lagrangian shock hydrodynamics simulations will fail to proceed past a certain time if the mesh is approaching tangling. A common solution is an Arbitrary Lagrangian Eulerian (ALE) form, in which the mesh is improved (remeshing) and the solution is remapped onto the improved mesh. The simplest remeshing techniques involve moving only the nodes of the mesh. More advanced remeshing techniques involve altering the mesh connectivity in portions of the domain in order to prevent tangling. Work has been done using Voronoi-based polygonal mesh generators and 2D quad/triangle mesh adaptation. Here, this paper presents the use of tetrahedral mesh adaptation methods as the remeshing step in an otherwise Lagrangian finite element shock hydrodynamics code called Alexa.
Supercomputing hardware is undergoing a period of significant change. In order to cope with the rapid pace of hardware and, in many cases, programming model innovation, we have developed the Kokkos Programming Model – a C++-based abstraction that permits performance portability across diverse architectures. Our experience has shown that the abstractions developed can significantly frustrate debugging and profiling activities because they break expected code proximity and layout assumptions. In this paper we present the Kokkos Profiling interface, a lightweight, suite of hooks to which debugging and profiling tools can attach to gain deep insights into the execution and data structure behaviors of parallel programs written to the Kokkos interface.
Park, Michael A.; Barral, Nicolas; Ibanez-Granados, Daniel A.; Kamenetskiy, Dmitry S.; Krakos, Joshua A.; Michal, Todd; Loseille, Adrien
Unstructured grid adaptation is a tool to control Computational Fluid Dynamics (CFD) discretization error. However, adaptive grid techniques have made limited impact on production analysis workflows where the control of discretization error is critical to obtaining reliable simulation results. Issues that prevent the use of adaptive grid methods are identified by applying unstructured grid adaptation methods to a series of benchmark cases. Once identified, these challenges to existing adaptive workflows can be addressed. Unstructured grid adaptation is evaluated for test cases described on the Turbulence Modeling Resource (TMR) web site, which documents uniform grid refinement of multiple schemes. The cases are turbulent flow over a Hemisphere Cylinder and an ONERA M6 Wing. Adaptive grid force and moment trajectories are shown for three integrated grid adaptation processes with Mach interpolation control and output error based metrics. The integrated grid adaptation process with a finite element (FE) discretization produced results consistent with uniform grid refinement of fixed grids. The integrated grid adaptation processes with finite volume schemes were slower to converge to the reference solution than the FE method. Metric conformity is documented on grid/metric snapshots for five grid adaptation mechanics implementations. These tools produce anisotropic boundary conforming grids requested by the adaptation process.
This report documents the ASC/ATDM Kokkos deliverable "Production Portable Dy- namic Task DAG Capability." This capability enables applications to create and execute a dynamic task DAG ; a collection of heterogeneous computational tasks with a directed acyclic graph (DAG) of "execute after" dependencies where tasks and their dependencies are dynamically created and destroyed as tasks execute. The Kokkos task scheduler executes the dynamic task DAG on the target execution resource; e.g. a multicore CPU, a manycore CPU such as Intel's Knights Landing (KNL), or an NVIDIA GPU. Several major technical challenges had to be addressed during development of Kokkos' Task DAG capability: (1) portability to a GPU with it's simplified hardware and micro- runtime, (2) thread-scalable memory allocation and deallocation from a bounded pool of memory, (3) thread-scalable scheduler for dynamic task DAG, (4) usability by applications.
Unstructured grid adaptation is a technology that holds the potential to improve the automation and accuracy of computational fluid dynamics and other computational disciplines. Difficulty producing the highly anisotropic elements necessary for simulation on complex curved geometries that satisfies a resolution request has limited this technology's widespread adoption. The Unstructured Grid Adaptation Working Group is an open gathering of researchers working on adapting simplicial meshes to conform to a metric field. Current members span a wide range of institutions including academia, industry, and national laboratories. The purpose of this group is to create a common basis for understanding and improving mesh adaptation. We present our first major contribution: a common set of benchmark cases, including input meshes and analytic metric specifications, that are publicly available to be used for evaluating any mesh adaptation code. We also present the results of several existing codes on these benchmark cases, to illustrate their utility in identifying key challenges common to all codes and important differences between available codes. Future directions are defined to expand this benchmark to mature the technology necessary to impact practical simulation workflows.