This is the manual for the TeMPI Shim library, whose goal is to facilitate scientific apps being loosely coupled through MPI. TeMPI Shim can be used to create and manage Message Passing Interface (MPI, see [MPI]) Communicators for Single Program, Multiple Data (SPMD, see [SPMD]) and Multiple Programs, Multiple Data (MPMD, see [MPMD]) workflows. MPI provides the MPI_APPNUM command number to each of the processes within its context. This number, starting at 0, is the application number being executed. In a case where 3 applications are being run in MPMD mode, there would be MPI_APPNUM values of 0, 1, and 2. TeMPI Shim creates intra- and inter-communicators between each pair of MPI_APPNUM values. In the aforementioned case, application 0 would have intra- and inter-communicators to speak to itself, application 1, and application 2. This is replicated for each of the applications. Additionally, TeMPI Shim creates intra- and inter-communicators for the first MPI rank of each application to directly communicate only with each other. Finally, TeMPI Shim creates its own copy of the default world communicator, i.e., MPI_COMM_WORLD. In the case where there is only a single application, it will have the communicators to only speak with itself. TeMPI Shim is useful in this case since it is considered good practice for MPI applications to copy the default world communicator and reference this copy (see [Duplicate World]_) anyways. Ultimately, it provides value independent of the number of applications present.
This report presents an assessment of immersed Eulerian-Lagrangian code-coupling techniques suitable for use in a broad range of mechanics applications. The coupling algorithm is based on an immersed finite element method that considers the Lagrangian and Eulerian overlap regions in the overall variational formulation. In this report the basic formulation details are presented followed by various aspects of the code-coupling algorithm using OpenIFEM as the Lagrangian/coupling framework. A series of representative test cases that illustrate the code-coupling algorithm are discussed. The current work provides an in-depth investigation into the immersed finite element method for the purposes of providing a rigorous coupling technique that is minimally invasive in the respective Eulerian and Lagrangian codes. A number of extensions to the base immersed finite element method have been examined. These extension include nodal and quadrature-based indicator functions, a Lagrangian volume-fraction calculation in regions of overlap, and the use of penalty constraints between the Lagrangian and Eulerian domains. A unique MPI-based coupling strategy that retains the independent MPI structure of each code has been demonstrated.
The Sandia National Laboratories (SNL) Large-Scale Computing Initiative (LSCI) milestone required running two parallel simulation codes at scale on the Trinity supercomputer at Los Alamos National Laboratory (LANL) to obtain presentation quality visualization results via in-situ methods. The two simulation codes used were Sandia Parallel Aerosciences Research Code (SPARC) and Nalu, both fluid dynamics codes developed at SNL. The codes were integrated with the ParaView Catalyst in-situ visualization library via the SNL developed Input Output SubSystem (IOSS). The LSCI milestone had a relatively short time-scale for completion of two months. During setup and execution of in-situ visualization for the milestone, there were several challenging issues in the areas of software builds, parallel startup-times, and in the a priori specification of visualizations. This paper will discuss the milestone activities and technical challenges encountered in its completion.
Sierra is an engineering mechanics simulation code suite supporting the Nation's Nuclear Weapons mission as well as other customers. It has explicit ties to Sandia National Labs' workflow, including geometry and meshing, design and optimization, and visualization. Distinguishing strengths include "application aware" development, scalability, SQA and V&V, multiple scales, and multi-physics coupling. This document is intended to help new and existing users of Sierra as a user manual and troubleshooting guide.
In this paper we describe the performance of two operations, gather and scatter. The operations we describe are simplified versions of those used in the implementation of sparse matrix libraries such as sparse-blas. Similar operations can also be found in benchmarks such as HPCG. Gather and scatter operations are memory load and store operations that are related to other memory operations such as Stream Triad and DAXPY, but have an additional dependence between memory loads that affects performance. We describe how the operations behave on current technology and identify features that enhance their performance. However, our description of hardware is general rather than specific to a single architecture.
