Publications

McGregor, Duncan A.; Love, Edward L.; Sirajuddin, David S.; Swan, Matthew S.; Collins, David I.; Pawlowski, Roger P.; Cartwright, Keith C.; Stafford, David S.

This is the user manual for EMPIRE, a simulation code for electromagnetics and plasma physics.

Crockatt, Michael M.; Shadid, John N.; Pawlowski, Roger P.; Conde, Sidafa C.; Mabuza, Sibu M.; Bonilla, Jesus B.

Abstract not provided.

Pawlowski, Roger P.

Abstract not provided.

Pawlowski, Roger P.

Abstract not provided.

Aaziz, Omar R.; Allan, Benjamin A.; Brandt, James M.; Cook, Jeanine C.; Devine, Karen D.; Elliott, James E.; Gentile, Ann C.; Hammond, Simon D.; Kelley, Brian M.; Lopatina, Lena L.; Moore, Stan G.; Olivier, Stephen L.; Pedretti, Kevin P.; Poliakoff, David Z.; Pawlowski, Roger P.; Regier, Phillip A.; Schmitz, Mark E.; Schwaller, Benjamin S.; Surjadidjaja, Vanessa S.; Swan, Matthew S.; Tucker, Nick T.; Tucker, Tom T.; Vaughan, Courtenay T.; Walton, Sara P.

Scientific applications run on high-performance computing (HPC) systems are critical for many national security missions within Sandia and the NNSA complex. However, these applications often face performance degradation and even failures that are challenging to diagnose. To provide unprecedented insight into these issues, the HPC Development, HPC Systems, Computational Science, and Plasma Theory & Simulation departments at Sandia crafted and completed their FY21 ASC Level 2 milestone entitled "Integrated System and Application Continuous Performance Monitoring and Analysis Capability." The milestone created a novel integrated HPC system and application monitoring and analysis capability by extending Sandia's Kokkos application portability framework, Lightweight Distributed Metric Service (LDMS) monitoring tool, and scalable storage, analysis, and visualization pipeline. The extensions to Kokkos and LDMS enable collection and storage of application data during run time, as it is generated, with negligible overhead. This data is combined with HPC system data within the extended analysis pipeline to present relevant visualizations of derived system and application metrics that can be viewed at run time or post run. This new capability was evaluated using several week-long, 290-node runs of Sandia's ElectroMagnetic Plasma In Realistic Environments ( EMPIRE ) modeling and design tool and resulted in 1TB of application data and 50TB of system data. EMPIRE developers remarked this capability was incredibly helpful for quickly assessing application health and performance alongside system state. In short, this milestone work built the foundation for expansive HPC system and application data collection, storage, analysis, visualization, and feedback framework that will increase total scientific output of Sandia's HPC users.

Tomas, Ignacio T.; Shadid, John N.; Crockatt, Michael M.; Pawlowski, Roger P.; Maier, Matthias M.; Guermond, Jean-Luc G.

This report summarizes the findings and outcomes of the LDRD-express project with title “Fluid models of charged species transport: numerical methods with mathematically guaranteed properties”. The primary motivation of this project was the computational/mathematical exploration of the ideas advanced aiming to improve the state-of-the-art on numerical methods for the one-fluid Euler-Poisson models and gain some understanding on the Euler-Maxwell model. Euler-Poisson and Euler-Maxwell, by themselves are not the most technically relevant PDE plasma-models. However, both of them are elementary building blocks of PDE-models used in actual technical applications and include most (if not all) of their mathematical difficulties. Outside the classical ideal MHD models, rigorous mathematical and numerical understanding of one-fluid models is still a quite undeveloped research area, and the treatment/understanding of boundary conditions is minimal (borderline non-existent) at this point in time. This report focuses primarily on bulk-behaviour of Euler-Poisson’s model, touching boundary conditions only tangentially.

Brandt, James M.; Cook, Jeanine C.; Aaziz, Omar R.; Allan, Benjamin A.; Devine, Karen D.; Elliott, James J.; Gentile, Ann C.; Hammond, Simon D.; Kelley, Brian M.; Lopatina, Lena L.; Moore, Stan G.; Olivier, Stephen L.; Pedretti, Kevin P.; Poliakoff, David Z.; Pawlowski, Roger P.; Regier, Phillip A.; Schmitz, Mark E.; Schwaller, Benjamin S.; Surjadidjaja, Vanessa S.; Swan, Matthew S.; Tucker, Tom T.; Tucker, Nick T.; Vaughan, Courtenay T.; Walton, Sara P.

Abstract not provided.

Crockatt, Michael M.; Shadid, John N.; Conde, Sidafa C.; Pawlowski, Roger P.; Mabuza, Sibu M.

Abstract not provided.

Communications in Computational Physics

Bettencourt, Matthew T.; Brown, Dominic A.S.; Cartwright, Keith L.; Cyr, Eric C.; Glusa, Christian A.; Lin, Paul T.; Moore, Stan G.; McGregor, Duncan A.O.; Pawlowski, Roger P.; Phillips, Edward G.; Roberts, Nathan V.; Wright, Steven A.; Maheswaran, Satheesh; Jones, John P.; Jarvis, Stephen A.

In this paper we introduce EMPIRE-PIC, a finite element method particle-in-cell (FEM-PIC) application developed at Sandia National Laboratories. The code has been developed in C++ using the Trilinos library and the Kokkos Performance Portability Framework to enable running on multiple modern compute architectures while only requiring maintenance of a single codebase. EMPIRE-PIC is capable of solving both electrostatic and electromagnetic problems in two- and three-dimensions to second-order accuracy in space and time. In this paper we validate the code against three benchmark problems - a simple electron orbit, an electrostatic Langmuir wave, and a transverse electromagnetic wave propagating through a plasma. We demonstrate the performance of EMPIRE-PIC on four different architectures: Intel Haswell CPUs, Intel's Xeon Phi Knights Landing, ARM Thunder-X2 CPUs, and NVIDIA Tesla V100 GPUs attached to IBM POWER9 processors. This analysis demonstrates scalability of the code up to more than two thousand GPUs, and greater than one hundred thousand CPUs.

Pawlowski, Roger P.; Phipps, Eric T.; Trott, Christian R.; Cyr, Eric C.; Shadid, John N.

Abstract not provided.

Crockatt, Michael M.; Shadid, John N.; Conde, Sidafa C.; Pawlowski, Roger P.; Mabuza, Sibu M.

Abstract not provided.

Phipps, Eric T.; Pawlowski, Roger P.; Trott, Christian R.

Abstract not provided.

Journal of Computational Physics

Mabuza, Sibusiso M.; Shadid, John N.; Cyr, Eric C.; Pawlowski, Roger P.; Kuzmin, Dmitri

In this work, a stabilized continuous Galerkin (CG) method for magnetohydrodynamics (MHD) is presented. Ideal, compressible inviscid MHD equations are discretized in space on unstructured meshes using piecewise linear or bilinear finite element bases to get a semi-discrete scheme. Stabilization is then introduced to the semi-discrete method in a strategy that follows the algebraic flux correction paradigm. This involves adding some artificial diffusion to the high order, semi-discrete method and mass lumping in the time derivative term. The result is a low order method that provides local extremum diminishing properties for hyperbolic systems. The difference between the low order method and the high order method is scaled element-wise using a limiter and added to the low order scheme. The limiter is solution dependent and computed via an iterative linearity preserving nodal variation limiting strategy. The stabilization also involves an optional consistent background high order dissipation that reduces phase errors. The resulting stabilized scheme is a semi-discrete method that can be applied to inviscid shock MHD problems and may be even extended to resistive and viscous MHD problems. To satisfy the divergence free constraint of the MHD equations, we add parabolic divergence cleaning to the system. Various time integration methods can be used to discretize the scheme in time. We demonstrate the robustness of the scheme by solving several shock MHD problems.

Miller, Sean M.; Cyr, Eric C.; Phillips, Edward G.; Pawlowski, Roger P.; Beckwith, Kristian B.; Bettencourt, Matthew T.; Cartwright, Keith C.; Shields, Sidney S.

Abstract not provided.

Pawlowski, Roger P.; Phipps, Eric T.

Abstract not provided.

Cyr, Eric C.; Shadid, John N.; Tuminaro, Raymond S.; Chacon, Luis C.; Pawlowski, Roger P.; Phillips, Edward G.; Miller, Sean M.

Abstract not provided.

Journal of Computational Physics

Miller, Sean M.; Cyr, E.C.; Shadid, John N.; Kramer, R.M.J.; Phillips, Edward G.; Conde, Sidafa C.; Pawlowski, Roger P.

Multi-fluid plasma models, where an electron fluid is modeled in addition to multiple ion and neutral species as well as the full set of Maxwell's equations, are useful for representing physics beyond the scope of classic MHD. This advantage presents challenges in appropriately dealing with electron dynamics and electromagnetic behavior characterized by the plasma and cyclotron frequencies and the speed of light. For physical systems, such as those near the MHD asymptotic regime, this requirement drastically increases runtimes for explicit time integration even though resolving fast dynamics may not be critical for accuracy. Implicit time integration methods, with efficient solvers, can help to step over fast time-scales that constrain stability, but do not strongly influence accuracy. As an extension, Implicit-explicit (IMEX) schemes provide an additional mechanism to choose which dynamics are evolved using an expensive implicit solve or resolved using a fast explicit solve. In this study, in addition to IMEX methods we also consider a physics compatible exact sequence spatial discretization. This combines nodal bases (H-Grad) for fluid dynamics with a set of vector bases (H-Curl and H-Div) for Maxwell's equations. This discretization allows for multi-fluid plasma modeling without violating Gauss' laws for the electric and magnetic fields. This initial study presents a discussion of the major elements of this formulation and focuses on demonstrating accuracy in the linear wave regime and in the MHD limit for both a visco-resistive and a dispersive ideal MHD problem.

Pawlowski, Roger P.

Abstract not provided.

Cyr, Eric C.; Miller, Sean M.; Pawlowski, Roger P.; Phillips, Edward G.; Beckwith, Kristian B.; Bettencourt, Matthew T.; Cartwright, Keith C.; Kramer, Richard M.; Roberts, Nathan V.; Shadid, John N.; Shields, Sidney S.

Abstract not provided.

Conde, Sidafa C.; Mabuza, Sibu M.; Shadid, John N.; Cyr, Eric C.; Smith, Thomas M.; Pawlowski, Roger P.

Abstract not provided.

Cyr, Eric C.; Miller, Sean M.; Phillips, Edward G.; Beckwith, Kristian B.; Bettencourt, Matthew T.; Cartwright, Keith C.; Kramer, Richard M.; Pawlowski, Roger P.; Roberts, Nathan V.; Shadid, John N.; Shields, Sidney S.

Abstract not provided.

Tuminaro, Raymond S.; Shadid, John N.; Cyr, Eric C.; Lin, Paul L.; Pawlowski, Roger P.; Phillips, Edward G.; Wiesner, Tobias A.; Adler, James A.; Benson, Tom B.; Chacon, Luis C.; Farrell, Patrick F.; Rappaport, Ari x.; MacLachlan, Scott M.; Hu, Jonathan J.; Berger-Vergiat, Luc B.; Glusa, Christian A.; Siefert, Christopher S.

Abstract not provided.

Lin, Paul L.; Shadid, John N.; Phillips, Edward G.; Pawlowski, Roger P.

Abstract not provided.

Pawlowski, Roger P.

Abstract not provided.

Lin, Paul L.; Shadid, John N.; Phillips, Edward G.; Pawlowski, Roger P.

Abstract not provided.