Publications

Hammond, Simon D.; Curry, Matthew J.; Davis, Kevin D.; Dang, Vinh Q.; Guba, Oksana G.; Hoekstra, Robert J.; Laros, James H.; Pedretti, Kevin P.; Poliakoff, David Z.; Rajamanickam, Sivasankaran R.; Trott, Christian R.; Younge, Andrew J.

Abstract not provided.

Hammond, Simon D.; Laros, James H.; Pedretti, Kevin P.; Younge, Andrew J.; Hoekstra, Robert J.

Abstract not provided.

Hughes, Clayton H.; Hammond, Simon D.; Zhang, Mengchi Z.; Liu, Yechen L.; Rogers, Tim R.; Hoekstra, Robert J.

Programmable accelerators have become commonplace in modern computing systems. Advances in programming models and the availability of unprecedented amounts of data have created a space for massively parallel accelerators capable of maintaining context for thousands of concurrent threads resident on-chip. These threads are grouped and interleaved on a cycle-by-cycle basis among several massively parallel computing cores. One path for the design of future supercomputers relies on an ability to model the performance of these massively parallel cores at scale. The SST framework has been proven to scale up to run simulations containing tens of thousands of nodes. A previous report described the initial integration of the open-source, execution-driven GPU simulator, GPGPU-Sim, into the SST framework. This report discusses the results of the integration and how to use the new GPU component in SST. It also provides examples of what it can be used to analyze and a correlation study showing how closely the execution matches that of a Nvidia V100 GPU when running kernels and mini-apps.

Hoekstra, Robert J.; Malone, C.M.; Montoya, D.R.; Ferencz, R.M.; Kuhl, A.L.; Hoekstra, R.J.; Wagner, J.W.

The review was conducted on May 8-9, 2017 at the University of Utah. Overall the review team was impressed with the work presented and found that the CCMSC had met or exceeded the Year 3 milestones. Specific details, comments, and recommendations are included in this document.

International Conference for High Performance Computing, Networking, Storage and Analysis, SC

Pedretti, Kevin P.; Younge, Andrew J.; Hammond, Simon D.; Laros, James H.; Curry, Matthew J.; Aguilar, Michael J.; Hoekstra, Robert J.; Brightwell, Ronald B.

Arm processors have been explored in HPC for several years, however there has not yet been a demonstration of viability for supporting large-scale production workloads. In this paper, we offer a retrospective on the process of bringing up Astra, the first Petascale supercomputer based on 64-bit Arm processors, and validating its ability to run production HPC applications. Through this process several immature technology gaps were addressed, including software stack enablement, Linux bugs at scale, thermal management issues, power management capabilities, and advanced container support. From this experience, several lessons learned are formulated that contributed to the successful deployment of Astra. These insights can be helpful to accelerate deploying and maturing other first-seen HPC technologies. With Astra now supporting many users running a diverse set of production applications at multi-thousand node scales, we believe this constitutes strong supporting evidence that Arm is a viable technology for even the largest-scale supercomputer deployments.

Hoekstra, Robert J.

Abstract not provided.

Hughes, Clayton H.; Hammond, Simon D.; Khairy, Mahmoud K.; Zhang, Mengchi Z.; Green, Roland G.; Rogers, Timothy R.; Hoekstra, Robert J.

Programmable accelerators have become commonplace in modern computing systems. Advances in programming models and the availability of massive amounts of data have created a space for massively parallel accelerators capable of maintaining context for thousands of concurrent threads resident on-chip. These threads are grouped and interleaved on a cycle-by-cycle basis among several massively parallel computing cores. One path for the design of future supercomputers relies on an ability to model the performance of these massively parallel cores at scale. The SST framework has been proven to scale up to run simulations containing tens of thousands of nodes. A previous report described the initial integration of the open-source, execution-driven GPU simulator, GPGPU-Sim, into the SST framework. This report discusses the results of the integration and how to use the new GPU component in SST. It also provides examples of what it can be used to analyze and a correlation study showing how closely the execution matches that of a Nvidia V100 GPU when running kernels and mini-apps.

Hughes, Clayton H.; Hammond, Simon D.; Voskuilen, Gwendolyn R.; Rodrigues, Arun; Hemmert, Karl S.; Hoekstra, Robert J.

Abstract not provided.

Hoekstra, Robert J.; Pedretti, Kevin P.; Hammond, Simon D.; Laros, James H.; Younge, Andrew J.; Lin, Paul L.; Vaughan, Courtney V.

Abstract not provided.

Hammond, Simon D.; Laros, James H.; Pedretti, Kevin P.; Younge, Andrew J.; Vaughan, Courtenay T.; Lin, Paul L.; Hoekstra, Robert J.

Abstract not provided.

Hughes, Clayton H.; Rodrigues, Arun; Voskuilen, Gwendolyn R.; Hemmert, Karl S.; Hammond, Simon D.; Hoekstra, Robert J.

Abstract not provided.

Hughes, Clayton H.; Hammond, Simon D.; Voskuilen, Gwendolyn R.; Rodrigues, Arun; Hemmert, Karl S.; Hoekstra, Robert J.

Abstract not provided.

Khairy, Mahmoud K.; Zhang, Mengchi Z.; Green, Roland G.; Hammond, Simon D.; Hoekstra, Robert J.; Rogers, Timothy R.; Hughes, Clayton H.

Programmable accelerators have become commonplace in modern computing systems. Advances in programming models and the availability of massive amounts of data have created a space for massively parallel acceleration where the context for thousands of concurrent threads are resident on-chip. These threads are grouped and interleaved on a cycle-by-cycle basis among several mas- sively parallel computing cores. The design of future supercomputers relies on an ability to model the performance of these massively parallel cores at scale. To address the need for a scalable, decentralized GPU model that can model large GPUs, chiplet- based GPUs and multi-node GPUs, this report details the first steps in integrating the open-source, execution driven GPGPU-Sim into the SST framework. The first stage of this project, creates two elements: a kernel scheduler SST element accepts work from SST CPU models and schedules it to an SM-collection element that performs cycle-by-cycle timing using SSTs Mem Hierarchy to model a flexible memory system.

Hughes, Clayton H.; Laros, James H.; Pedretti, Kevin P.; Hammond, Simon D.; Younge, Andrew J.; Hoekstra, Robert J.

Abstract not provided.

Hughes, Clayton H.; Laros, James H.; Pedretti, Kevin P.; Hammond, Simon D.; Younge, Andrew J.; Hoekstra, Robert J.

Abstract not provided.

Hoekstra, Robert J.; Bartlett, Roscoe B.; Hammond, Simon D.; Cook, Jeanine C.; Dinge, Dennis D.; Frye, Joseph R.; Hughes, Clayton H.; Lin, Paul L.; Vaughan, Courtenay T.; Hammond, Simon D.

Abstract not provided.

Alvin, Kenneth F.; Laros, James H.; Hoekstra, Robert J.; Collis, Samuel S.; Lujan, Jim L.

Abstract not provided.

Hammond, Simon D.; Laros, James H.; Younge, Andrew J.; Pedretti, Kevin P.; Hoekstra, Robert J.

Abstract not provided.

Hammond, Simon D.; Hoekstra, Robert J.

Abstract not provided.

Laros, James H.; Pedretti, Kevin P.; Hammond, Simon D.; Aguilar, Michael J.; Curry, Matthew L.; Grant, Ryan E.; Hoekstra, Robert J.; Klundt, Ruth A.; Monk, Stephen T.; Ogden, Jeffry B.; Olivier, Stephen L.; Scott, Randall D.; Ward, Harry L.; Younge, Andrew J.

The Vanguard program informally began in January 2017 with the submission of a white pa- per entitled "Sandia's Vision for a 2019 Arm Testbed" to NNSA headquarters. The program proceeded in earnest in May 2017 with an announcement by Doug Wade (Director, Office of Advanced Simulation and Computing and Institutional R&D at NNSA) that Sandia Na- tional Laboratories (Sandia) would host the first Advanced Architecture Prototype platform based on the Arm architecture. In August 2017, Sandia formed a Tri-lab team chartered to develop a robust HPC software stack for Astra to support the Vanguard program goal of demonstrating the viability of Arm in supporting ASC production computing workloads. This document describes the high-level Vanguard program goals, the Vanguard-Astra project acquisition plan and procurement up to contract placement, the initial software stack environment planned for the Vanguard-Astra platform (Astra), a description of how the communities of users will utilize the platform during the transition from the open network to the classified network, and initial performance results.

Laros, James H.; Pedretti, Kevin P.; Hammond, Simon D.; Aguilar, Michael J.; Curry, Matthew L.; Grant, Ryan E.; Hoekstra, Robert J.; Klundt, Ruth A.; Monk, Stephen T.; Ogden, Jeffry B.; Olivier, Stephen L.; Scott, Randall D.; Ward, Harry L.; Younge, Andrew J.

The Vanguard program informally began in January 2017 with the submission of a white pa- per entitled "Sandia's Vision for a 2019 Arm Testbed" to NNSA headquarters. The program proceeded in earnest in May 2017 with an announcement by Doug Wade (Director, Office of Advanced Simulation and Computing and Institutional R&D at NNSA) that Sandia Na- tional Laboratories (Sandia) would host the first Advanced Architecture Prototype platform based on the Arm architecture. In August 2017, Sandia formed a Tri-lab team chartered to develop a robust HPC software stack for Astra to support the Vanguard program goal of demonstrating the viability of Arm in supporting ASC production computing workloads. This document describes the high-level Vanguard program goals, the Vanguard-Astra project acquisition plan and procurement up to contract placement, the initial software stack environment planned for the Vanguard-Astra platform (Astra), a description of how the communities of users will utilize the platform during the transition from the open network to the classified network, and initial performance results.

Hoekstra, Robert J.; Hungerford, Aimee H.; Montoya, David M.; Ferencz, Robert F.; Kuhl, Alan K.; Ruggirello, Kevin P.

Abstract not provided.

Vaughan, Courtenay T.; Cook, Jeanine C.; Benner, R.E.; Dinge, Dennis D.; Lin, Paul L.; Hughes, Clayton H.; Hoekstra, Robert J.; Hammond, Simon D.

Abstract not provided.

Hammond, Simon D.; Rodrigues, Arun; Voskuilen, Gwendolyn R.; Hemmert, Karl S.; Levenhagen, Michael J.; Hughes, Clayton H.; Hoekstra, Robert J.

Abstract not provided.

Vaughan, Courtenay T.; Hammond, Simon D.; Dinge, Dennis D.; Lin, Paul L.; Hughes, Clayton H.; Benner, R.E.; Cook, Jeanine C.; Pase, Douglas M.; Hoekstra, Robert J.

Abstract not provided.

Laros, James H.; Alvin, Kenneth F.; Hoekstra, Robert J.; Pedretti, Kevin P.; Hammond, Simon D.

Abstract not provided.

Hoekstra, Robert J.

Abstract not provided.

Hammond, Simon D.; Trott, Christian R.; Ibanez-Granados, Daniel A.; Edwards, Harold C.; Sunderland, Daniel S.; Ellingwood, Nathan D.; Brandt, James M.; Gentile, Ann C.; Cook, Jeanine C.; Hoekstra, Robert J.

Abstract not provided.

Hammond, Simon D.; Hughes, Clayton H.; Awad, Amro A.; Voskuilen, Gwendolyn R.; Rodrigues, Arun; Hemmert, Karl S.; Levenhagen, Michael J.; Hoekstra, Robert J.

Abstract not provided.

Hughes, Clayton H.; Awad, Amro A.; Hammond, Simon D.; Rodrigues, Arun; Hemmert, Karl S.; Hoekstra, Robert J.

Abstract not provided.

Hammond, Simon D.; Vaughan, Courtenay T.; Dinge, Dennis D.; Lin, Paul L.; Benner, R.E.; Hughes, Clayton H.; Trott, Christian R.; Cook, Jeanine C.; Hoekstra, Robert J.

Abstract not provided.

Hammond, Simon D.; Rodrigues, Arun; Hemmert, Karl S.; Voskuilen, Gwendolyn R.; Hughes, Clayton H.; Levenhagen, Michael J.; Hoekstra, Robert J.; Ang, James A.

Abstract not provided.

Hammond, Simon D.; Hoekstra, Robert J.; Rodrigues, Arun; Ang, James A.

Abstract not provided.

Hoekstra, Robert J.

Abstract not provided.

ACM International Conference Proceeding Series

Awad, Amro A.; Hammond, Simon D.; Hughes, Clayton H.; Rodrigues, Arun; Hemmert, Karl S.; Hoekstra, Robert J.

DRAM scalability is becoming more challenging, pushing the focus of the research community towards alternative memory technologies. Many emerging non-volatile memory (NVM) devices are proving themselves to be good candidates to replace DRAM in the coming years. For example, the recently announced 3D-XPoint memory by Intel/Micron promises latencies that are comparable to DRAM, while being non-volatile and much more dense. While emerging NVMs can be fabricated in different form factors, the most promising (from a performance perspective) are NVM-based DIMMs. Unfortunately, there is a shortage of studies that explore the design options for NVM-based DIMMs. Because of the read and write asymmetries in both power consumption and latency, as well as limited write endurance, which often requires wear-leveling techniques, NVMs require a specialized controller. The fact that future on-die memory controllers are expected to handle different memory technologies pushes future hardware towards on-DIMM controllers. In this paper, we propose an architectural model for NVM-based DIMMs with internal controllers, explore their design space, evaluate different optimizations and reach out to several architectural suggestions. Finally, we make our model publicly available and integrate it with a widely used architectural simulator.

Ang, James A.; Brightwell, Ronald B.; Hammond, Simon D.; Hemmert, Karl S.; Hoekstra, Robert J.; Laros, James H.; Pedretti, Kevin P.; Rodrigues, Arun

Abstract not provided.

Hoekstra, Robert J.; Hammond, Simon D.; Hemmert, Karl S.; Gentile, Ann C.; Oldfield, Ron A.; Lang, Mike L.; Martin, Steve M.

The presentation documented the technical approach of the team and summary of the results with sufficient detail to demonstrate both the value and the completion of the milestone. A separate SAND report was also generated with more detail to supplement the presentation.

Hoekstra, Robert J.; Hammond, Simon D.; Richards, David F.; Bergen, Ben B.

This milestone is a tri-lab deliverable supporting ongoing Co-Design efforts impacting applications in the Integrated Codes (IC) program element Advanced Technology Development and Mitigation (ATDM) program element. In FY14, the trilabs looked at porting proxy application to technologies of interest for ATS procurements. In FY15, a milestone was completed evaluating proxy applications in multiple programming models and in FY16, a milestone was completed focusing on the migration of lessons learned back into production code development. This year, the co-design milestone focuses on extracting the knowledge gained and/or code revisions back into production applications.

Awad, Amro A.; Hammond, Simon D.; Hughes, Clayton H.; Rodrigues, Arun; Hemmert, Karl S.; Hoekstra, Robert J.

Abstract not provided.

Ang, James A.; Hammond, Simon D.; Hoekstra, Robert J.; Laros, James H.; Rodrigues, Arun

Abstract not provided.

Vaughan, Courtenay T.; Hammond, Simon D.; Dinge, Dennis D.; Lin, Paul L.; Pase, Douglas M.; Cook, Jeanine C.; Trott, Christian R.; Hughes, Clayton H.; Hoekstra, Robert J.

Abstract not provided.

Vaughan, Courtenay T.; Hammond, Simon D.; Dinge, Dennis D.; Lin, Paul L.; Pase, Douglas M.; Trott, Christian R.; Cook, Jeanine C.; Hughes, Clayton H.; Hoekstra, Robert J.

Abstract not provided.

Hoekstra, Robert J.; Hammond, Simon D.; Richards, David F.; McCormick, Patrick M.

Abstract not provided.

Hoekstra, Robert J.; Leon, Edgar A.; Bergen, Ben B.; Carribault, Patrick C.

Abstract not provided.

Vaughan, Courtenay T.; Dinge, Dennis D.; Lin, Paul L.; Hammond, Simon D.; Pase, Douglas M.; Benner, Douglas E.; Cook, Jeanine C.; Hoekstra, Robert J.

Abstract not provided.

Hammond, Simon D.; Trott, Christian R.; Vaughan, Courtenay T.; Dinge, Dennis D.; Lin, Paul L.; Pase, Douglas M.; Benner, R.E.; Cook, Jeanine C.; Hoekstra, Robert J.

Abstract not provided.

Awad, Amro A.; Voskuilen, Gwendolyn R.; Rodrigues, Arun; Hammond, Simon D.; Hoekstra, Robert J.; Hughes, Clayton H.

DRAM technology is the main building block of main memory, however, DRAM scaling is becoming very challenging. The main issues for DRAM scaling are the increasing error rates with each new generation, the geometric and physical constraints of scaling the capacitor part of the DRAM cells, and the high power consumption caused by the continuous need for refreshing cell values. At the same time, emerging Non- Volatile Memory (NVM) technologies, such as Phase-Change Memory (PCM), are emerging as promising replacements for DRAM. NVMs, when compared to current technologies e.g., NAND-based ash, have latencies comparable to DRAM. Additionally, NVMs are non-volatile, which eliminates the need for refresh power and enables persistent memory applications. Finally, NVMs have promising densities and the potential for multi-level cell (MLC) storage.

Hoekstra, Robert J.; Rodrigues, Arun

Abstract not provided.

Hammond, Simon D.; Vaughan, Courtenay T.; Dinge, Dennis D.; Lin, Paul L.; Pase, Douglas M.; Cook, Jeanine C.; Hoekstra, Robert J.

Abstract not provided.

Laros, James H.; Hoekstra, Robert J.; Hammond, Simon D.; Alvin, Kenneth F.

Abstract not provided.

Hoekstra, Robert J.

Abstract not provided.

Marinella, Matthew J.; Hoekstra, Robert J.

Abstract not provided.

Hammond, Simon D.; Trott, Christian R.; Lin, Paul L.; Vaughan, Courtenay T.; Cook, Jeanine C.; Dinge, Dennis D.; Pase, Douglas M.; Benner, R.E.; Hoekstra, Robert J.

Abstract not provided.

Hoekstra, Robert J.; Ruggirello, Kevin P.

Abstract not provided.

Ang, James A.; Barrett, Richard F.; Benner, R.E.; Burke, Daniel B.; Chan, Cy P.; Cook, Jeanine C.; Daley, Christopher D.; Donofrio, Dave D.; Hammond, Simon D.; Hemmert, Karl S.; Hoekstra, Robert J.; Ibrahim, Khaled I.; Kelly, Suzanne M.; Le, Hoang L.; Leung, Vitus J.; Michelogiannakis, George M.; Resnick, David R.; Rodrigues, Arun; Shalf, John S.; Stark, Dylan S.; Unat, D.U.; Wright, Nick W.; Voskuilen, Gwendolyn R.

Machine Models and Proxy Architectures for Exascale Computing Version 2.0 Prepared by Sandia National Laboratories Albuquerque, New Mexico 87185 and Livermore, California 94550 Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-AC04-94AL85000. Approved for public release; further dissemination unlimited. Issued by Sandia National Laboratories, operated for the United States Department of Energy by Sandia Corporation. NOTICE: This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government, nor any agency thereof, nor any of their employees, nor any of their contractors, subcontractors, or their employees, make any warranty, express or implied, or assume any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or rep- resent that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government, any agency thereof, or any of their contractors or subcontractors. The views and opinions expressed herein do not necessarily state or reflect those of the United States Government, any agency thereof, or any of their contractors. Printed in the United States of America. This report has been reproduced directly from the best available copy. Available to DOE and DOE contractors from U.S. Department of Energy Office of Scientific and Technical Information P.O. Box 62 Oak Ridge, TN 37831 Telephone: (865) 576-8401 Facsimile: (865) 576-5728 E-Mail: reports@adonis.osti.gov Online ordering: http://www.osti.gov/bridge Available to the public from U.S. Department of Commerce National Technical Information Service 5285 Port Royal Rd Springfield, VA 22161 Telephone: (800) 553-6847 Facsimile: (703) 605-6900 E-Mail: orders@ntis.fedworld.gov Online ordering: http://www.ntis.gov/help/ordermethods.asp?loc=7-4-0#online D E P A R T M E N T O F E N E R G Y * * U N I T E D S T A T E S O F A M E R I C A SAND2016-6049 Unlimited Release Printed Abstract Machine Models and Proxy Architectures for Exascale Computing Version 2.0 J.A. Ang 1 , R.F. Barrett 1 , R.E. Benner 1 , D. Burke 2 , C. Chan 2 , J. Cook 1 , C.S. Daley 2 , D. Donofrio 2 , S.D. Hammond 1 , K.S. Hemmert 1 , R.J. Hoekstra 1 , K. Ibrahim 2 , S.M. Kelly 1 , H. Le, V.J. Leung 1 , G. Michelogiannakis 2 , D.R. Resnick 1 , A.F. Rodrigues 1 , J. Shalf 2 , D. Stark, D. Unat, N.J. Wright 2 , G.R. Voskuilen 1 1 1 Sandia National Laboratories, P.O. Box 5800, Albuquerque, New Mexico 87185-MS 1319 2 Lawrence Berkeley National Laboratory, Berkeley, California Abstract To achieve exascale computing, fundamental hardware architectures must change. The most sig- nificant consequence of this assertion is the impact on the scientific and engineering applications that run on current high performance computing (HPC) systems, many of which codify years of scientific domain knowledge and refinements for contemporary computer systems. In order to adapt to exascale architectures, developers must be able to reason about new hardware and deter- mine what programming models and algorithms will provide the best blend of performance and energy efficiency into the future. While many details of the exascale architectures are undefined, an abstract machine model is designed to allow application developers to focus on the aspects of the machine that are important or relevant to performance and code structure. These models are intended as communication aids between application developers and hardware architects during the co-design process. We use the term proxy architecture to describe a parameterized version of an abstract machine model, with the parameters added to elucidate potential speeds and capacities of key hardware components. These more detailed architectural models are formulated to enable discussion between the developers of analytic models and simulators and computer hardware archi- tects. They allow for application performance analysis and hardware optimization opportunities. In this report our goal is to provide the application development community with a set of mod- els that can help software developers prepare for exascale. In addition, through the use of proxy architectures, we can enable a more concrete exploration of how well new and evolving applica- tion codes map onto future architectures. This second version of the document addresses system scale considerations and provides a system-level abstract machine model with proxy architecture information.

Vaughan, Courtenay T.; Dinge, Dennis D.; Lin, Paul L.; Hammond, Simon D.; Cook, Jeanine C.; Trott, Christian R.; Agelastos, Anthony M.; Pase, Douglas M.; Benner, R.E.; Rajan, Mahesh R.; Hoekstra, Robert J.

Abstract not provided.

Hemmert, Karl S.; Rajan, Mahesh R.; Hoekstra, Robert J.; Dawson, Shawn D.; Vigil, Manuel V.; Grunau, Daryl G.; Lujan, James L.; Morton, David P.; Nam, Hai A.; Peltz, Paul Jr.; Torrez, Alfred T.; Wright, Cornell W.; Glass, Micheal W.; Hammond, Simon D.

Abstract not provided.

Hemmert, Karl S.; Rajan, Mahesh R.; Hoekstra, Robert J.; Dawson, Shawn D.; Vigil, Manuel V.; Grunau, Daryl G.; Lujan, James L.; Morton, David P.; Nam, Hai A.; Peltz, Paul Jr.; Torrez, Alfred T.; Wright, Cornell W.; Glass, Micheal W.; Hammond, Simon D.

Abstract not provided.

Hemmert, Karl S.; Rajan, Mahesh R.; Hoekstra, Robert J.; Dawson, Shawn D.; Vigil, Manuel V.; Grunau, Daryl G.; Lujan, James L.; Morton, David P.; Nam, Hai A.; Peltz, Paul Jr.; Torrez, Alfred T.; Wright, Cornell W.

Abstract not provided.

Vaughan, Courtenay T.; Dinge, Dennis D.; Lin, Paul L.; Hammond, Simon D.; Cook, Jeanine C.; Trott, Christian R.; Agelastos, Anthony M.; Pase, Douglas M.; Benner, R.E.; Rajan, Mahesh R.; Hoekstra, Robert J.; Pierson, Kendall H.

Abstract not provided.

Hoekstra, Robert J.

Abstract not provided.

Trott, Christian R.; Hammond, Simon D.; Dinge, Dennis D.; Lin, Paul L.; Vaughan, Courtenay T.; Cook, Jeanine C.; Rajan, Mahesh R.; Edwards, Harold C.; Hoekstra, Robert J.

For the FY15 ASC L2 Trilab Codesign milestone Sandia National Laboratories performed two main studies. The first study investigated three topics (performance, cross-platform portability and programmer productivity) when using OpenMP directives and the RAJA and Kokkos programming models available from LLNL and SNL respectively. The focus of this first study was the LULESH mini-application developed and maintained by LLNL. In the coming sections of the report the reader will find performance comparisons (and a demonstration of portability) for a variety of mini-application implementations produced during this study with varying levels of optimization. Of note is that the implementations utilized including optimizations across a number of programming models to help ensure claims that Kokkos can provide native-class application performance are valid. The second study performed during FY15 is a performance assessment of the MiniAero mini-application developed by Sandia. This mini-application was developed by the SIERRA Thermal-Fluid team at Sandia for the purposes of learning the Kokkos programming model and so is available in only a single implementation. For this report we studied its performance and scaling on a number of machines with the intent of providing insight into potential performance issues that may be experienced when similar algorithms are deployed on the forthcoming Trinity ASC ATS platform.

Hoekstra, Robert J.

Abstract not provided.

Cook, Jeanine C.; Edwards, Harold C.; Dinge, Dennis D.; Glass, Micheal W.; Hammond, Simon D.; Hoekstra, Robert J.; Lin, Paul L.; Rajan, Mahesh R.; Trott, Christian R.; Vaughan, Courtenay T.

Abstract not provided.

Hoekstra, Robert J.

Abstract not provided.

Hoekstra, Robert J.; Summers, Randall M.

Abstract not provided.

Hoekstra, Robert J.; Barrett, Richard F.; howell, louis h.; daniel, david d.

This milestone was the 2nd in a series of Tri-Lab Co-Design L2 milestones supporting ‘Co-Design’ efforts in the ASC program. It is a crucial step towards evaluating the effectiveness of proxy applications in exploring code performance on next generation architectures. All three labs evaluated the performance of 2 proxy applications on modern architectures and/or testbeds for pre-production hardware. The results are captured in this document as well as annotated presentations from all 3 laboratories.

Hoekstra, Robert J.

Abstract not provided.

Hoekstra, Robert J.

Abstract not provided.

Hemstad, Jacob H.; Heroux, Michael A.; Hoekstra, Robert J.

Abstract not provided.

Hemstad, Jacob H.; Heroux, Michael A.; Hoekstra, Robert J.

Abstract not provided.

Glass, Micheal W.; Hoekstra, Robert J.

Abstract not provided.

Hoekstra, Robert J.

Abstract not provided.

Schoenwald, David A.; Richardson, Bryan T.; Riehm, Andrew C.; Wolfenbarger, Paul W.; Adams, Brian M.; Reno, Matthew J.; Hansen, Clifford H.; Oldfield, Ron A.; Stamp, Jason E.; Stein, Joshua S.; Hoekstra, Robert J.; Munoz-Ramos, Karina M.; McLendon, William C.; Russo, Thomas V.; Phillips, Laurence R.

Design and operation of the electric power grid (EPG) relies heavily on computational models. High-fidelity, full-order models are used to study transient phenomena on only a small part of the network. Reduced-order dynamic and power flow models are used when analysis involving thousands of nodes are required due to the computational demands when simulating large numbers of nodes. The level of complexity of the future EPG will dramatically increase due to large-scale deployment of variable renewable generation, active load and distributed generation resources, adaptive protection and control systems, and price-responsive demand. High-fidelity modeling of this future grid will require significant advances in coupled, multi-scale tools and their use on high performance computing (HPC) platforms. This LDRD report demonstrates SNL's capability to apply HPC resources to these 3 tasks: (1) High-fidelity, large-scale modeling of power system dynamics; (2) Statistical assessment of grid security via Monte-Carlo simulations of cyber attacks; and (3) Development of models to predict variability of solar resources at locations where little or no ground-based measurements are available.

Keiter, Eric R.; Thornquist, Heidi K.; Hoekstra, Robert J.; Russo, Thomas V.; Schiek, Richard S.

Abstract not provided.

Hoekstra, Robert J.

Abstract not provided.

Keiter, Eric R.; Santarelli, Keith R.; Hoekstra, Robert J.; Russo, Thomas V.; Schiek, Richard S.; Mei, Ting M.; Thornquist, Heidi K.; Pawlowski, Roger P.; Coffey, Todd S.

The Xyce Parallel Electronic Simulator has been written to support, in a rigorous manner, the simulation needs of the Sandia National Laboratories electrical designers. Specific requirements include, among others, the ability to solve extremely large circuit problems by supporting large-scale parallel computing platforms, improved numerical performance and object-oriented code design and implementation. The Xyce release notes describe: Hardware and software requirements New features and enhancements Any defects fixed since the last release Current known defects and defect workarounds For up-to-date information not available at the time these notes were produced, please visit the Xyce web page at http://www.cs.sandia.gov/xyce.

Journal of Computational Physics

Lin, Paul T.; Shadid, John N.; Sala, Marzio; Tuminaro, Raymond S.; Hennigan, Gary L.; Hoekstra, Robert J.

In this study results are presented for the large-scale parallel performance of an algebraic multilevel preconditioner for solution of the drift-diffusion model for semiconductor devices. The preconditioner is the key numerical procedure determining the robustness, efficiency and scalability of the fully-coupled Newton-Krylov based, nonlinear solution method that is employed for this system of equations. The coupled system is comprised of a source term dominated Poisson equation for the electric potential, and two convection-diffusion-reaction type equations for the electron and hole concentration. The governing PDEs are discretized in space by a stabilized finite element method. Solution of the discrete system is obtained through a fully-implicit time integrator, a fully-coupled Newton-based nonlinear solver, and a restarted GMRES Krylov linear system solver. The algebraic multilevel preconditioner is based on an aggressive coarsening graph partitioning of the nonzero block structure of the Jacobian matrix. Representative performance results are presented for various choices of multigrid V-cycles and W-cycles and parameter variations for smoothers based on incomplete factorizations. Parallel scalability results are presented for solution of up to 108 unknowns on 4096 processors of a Cray XT3/4 and an IBM POWER eServer system. © 2009 Elsevier Inc. All rights reserved.

Keiter, Eric R.; Thornquist, Heidi K.; Day, David M.; Boman, Erik G.; Hoekstra, Robert J.

Abstract not provided.

Hoekstra, Robert J.; Hennigan, Gary L.; Pawlowski, Roger P.; Phipps, Eric T.; Shadid, John N.; Musson, Lawrence M.; Keiter, Eric R.

Abstract not provided.

Shadid, John N.; Hoekstra, Robert J.

Abstract not provided.

Rutherford, Brian M.; Romero, Vicente J.; Hoekstra, Robert J.; Castro, Joseph P.

Abstract not provided.

Lin, Paul L.; Shadid, John N.; Hoekstra, Robert J.; Hennigan, Gary L.

Abstract not provided.

Hoekstra, Robert J.; Thornquist, Heidi K.; Day, David M.; Boman, Erik G.

Abstract not provided.

Bartlett, Roscoe B.; Hoekstra, Robert J.

Automatic differentiation (AD) is useful in transient sensitivity analysis of a computational simulation of a bipolar junction transistor subject to radiation damage. We used forward-mode AD, implemented in a new Trilinos package called Sacado, to compute analytic derivatives for implicit time integration and forward sensitivity analysis. Sacado addresses element-based simulation codes written in C++ and works well with forward sensitivity analysis as implemented in the Trilinos time-integration package Rythmos. The forward sensitivity calculation is significantly more efficient and robust than finite differencing.

Hoekstra, Robert J.; Castro, Joseph P.; Shadid, John N.; Fixel, Deborah A.

A code, Charon, is described which simulates the effects that neutron damage has on silicon semiconductor devices. The code uses a stabilized, finite-element discretization of the semiconductor drift-diffusion equations. The mathematical model used to simulate semiconductor devices in both normal and radiation environments will be described. Modeling of defect complexes is accomplished by adding an additional drift-diffusion equation for each of the defect species. Additionally, details are given describing how Charon can efficiently solve very large problems using modern parallel computers. Comparison between Charon and experiment will be given, as well as comparison with results from commercially-available TCAD codes.

Lin, Paul L.; Shadid, John N.; Hoekstra, Robert J.; Hennigan, Gary L.

Abstract not provided.

Hoekstra, Robert J.; Keiter, Eric R.; Phipps, Eric T.

Abstract not provided.

Tuminaro, Raymond S.; Shadid, John N.; Hoekstra, Robert J.; Hennigan, Gary L.

Abstract not provided.

Keiter, Eric R.; Hoekstra, Robert J.; Coffey, Todd S.; Heroux, Michael A.; Pawlowski, Roger P.; Hutchinson, Scott A.

Abstract not provided.

Hutchinson, Scott A.; Keiter, Eric R.; Hoekstra, Robert J.; Russo, Thomas V.; Rankin, Eric R.; Pawlowski, Roger P.; Fixel, Deborah A.; Schiek, Richard S.; Bogdan, Carolyn W.

This manual describes the use of theXyceParallel Electronic Simulator.Xycehasbeen designed as a SPICE-compatible, high-performance analog circuit simulator, andhas been written to support the simulation needs of the Sandia National Laboratorieselectrical designers. This development has focused on improving capability over thecurrent state-of-the-art in the following areas:%04Capability to solve extremely large circuit problems by supporting large-scale par-allel computing platforms (up to thousands of processors). Note that this includessupport for most popular parallel and serial computers.%04Improved performance for all numerical kernels (e.g., time integrator, nonlinearand linear solvers) through state-of-the-art algorithms and novel techniques.%04Device models which are specifically tailored to meet Sandia's needs, includingmany radiation-aware devices.3 XyceTMUsers' Guide%04Object-oriented code design and implementation using modern coding practicesthat ensure that theXyceParallel Electronic Simulator will be maintainable andextensible far into the future.Xyceis a parallel code in the most general sense of the phrase - a message passingparallel implementation - which allows it to run efficiently on the widest possible numberof computing platforms. These include serial, shared-memory and distributed-memoryparallel as well as heterogeneous platforms. Careful attention has been paid to thespecific nature of circuit-simulation problems to ensure that optimal parallel efficiencyis achieved as the number of processors grows.The development ofXyceprovides a platform for computational research and de-velopment aimed specifically at the needs of the Laboratory. WithXyce, Sandia hasan %22in-house%22 capability with which both new electrical (e.g., device model develop-ment) and algorithmic (e.g., faster time-integration methods, parallel solver algorithms)research and development can be performed. As a result,Xyceis a unique electricalsimulation capability, designed to meet the unique needs of the laboratory.4 XyceTMUsers' GuideAcknowledgementsThe authors would like to acknowledge the entire Sandia National Laboratories HPEMS(High Performance Electrical Modeling and Simulation) team, including Steve Wix, CarolynBogdan, Regina Schells, Ken Marx, Steve Brandon and Bill Ballard, for their support onthis project. We also appreciate very much the work of Jim Emery, Becky Arnold and MikeWilliamson for the help in reviewing this document.Lastly, a very special thanks to Hue Lai for typesetting this document with LATEX.TrademarksThe information herein is subject to change without notice.Copyrightc 2002-2003 Sandia Corporation. All rights reserved.XyceTMElectronic Simulator andXyceTMtrademarks of Sandia Corporation.Orcad, Orcad Capture, PSpice and Probe are registered trademarks of Cadence DesignSystems, Inc.Silicon Graphics, the Silicon Graphics logo and IRIX are registered trademarks of SiliconGraphics, Inc.Microsoft, Windows and Windows 2000 are registered trademark of Microsoft Corporation.Solaris and UltraSPARC are registered trademarks of Sun Microsystems Corporation.Medici, DaVinci and Taurus are registered trademarks of Synopsys Corporation.HP and Alpha are registered trademarks of Hewlett-Packard company.Amtec and TecPlot are trademarks of Amtec Engineering, Inc.Xyce's expression library is based on that inside Spice 3F5 developed by the EECS De-partment at the University of California.All other trademarks are property of their respective owners.ContactsBug Reportshttp://tvrusso.sandia.gov/bugzillaEmailxyce-support%40sandia.govWorld Wide Webhttp://www.cs.sandia.gov/xyce5 XyceTMUsers' GuideThis page is left intentionally blank6

Pawlowski, Roger P.; Keiter, Eric R.; Hoekstra, Robert J.; Hutchinson, Scott A.; Russo, Thomas V.

Abstract not provided.

Keiter, Eric R.; Hutchinson, Scott A.; Hoekstra, Robert J.; Russo, Thomas V.; Rankin, Eric R.; Pawlowski, Roger P.; Wix, Steven D.; Fixel, Deborah A.

This manual describes the use of the Xyce Parallel Electronic Simulator. Xyce has been designed as a SPICE-compatible, high-performance analog circuit simulator capable of simulating electrical circuits at a variety of abstraction levels. Primarily, Xyce has been written to support the simulation needs of the Sandia National Laboratories electrical designers. This development has focused on improving capability the current state-of-the-art in the following areas: {sm_bullet} Capability to solve extremely large circuit problems by supporting large-scale parallel computing platforms (up to thousands of processors). Note that this includes support for most popular parallel and serial computers. {sm_bullet} Improved performance for all numerical kernels (e.g., time integrator, nonlinear and linear solvers) through state-of-the-art algorithms and novel techniques. {sm_bullet} Device models which are specifically tailored to meet Sandia's needs, including many radiation-aware devices. {sm_bullet} A client-server or multi-tiered operating model wherein the numerical kernel can operate independently of the graphical user interface (GUI). {sm_bullet} Object-oriented code design and implementation using modern coding practices that ensure that the Xyce Parallel Electronic Simulator will be maintainable and extensible far into the future. Xyce is a parallel code in the most general sense of the phrase - a message passing of computing platforms. These include serial, shared-memory and distributed-memory parallel implementation - which allows it to run efficiently on the widest possible number parallel as well as heterogeneous platforms. Careful attention has been paid to the specific nature of circuit-simulation problems to ensure that optimal parallel efficiency is achieved as the number of processors grows. One feature required by designers is the ability to add device models, many specific to the needs of Sandia, to the code. To this end, the device package in the Xyce These input formats include standard analytical models, behavioral models look-up Parallel Electronic Simulator is designed to support a variety of device model inputs. tables, and mesh-level PDE device models. Combined with this flexible interface is an architectural design that greatly simplifies the addition of circuit models. One of the most important feature of Xyce is in providing a platform for computational research and development aimed specifically at the needs of the Laboratory. With Xyce, Sandia now has an 'in-house' capability with which both new electrical (e.g., device model development) and algorithmic (e.g., faster time-integration methods) research and development can be performed. Ultimately, these capabilities are migrated to end users.

Keiter, Eric R.; Hutchinson, Scott A.; Hoekstra, Robert J.; Russo, Thomas V.; Rankin, Eric R.; Pawlowski, Roger P.; Fixel, Deborah A.; Wix, Steven D.

This document is a reference guide to the Xyce Parallel Electronic Simulator, and is a companion document to the Xyce Users' Guide. The focus of this document is (to the extent possible) exhaustively list device parameters, solver options, parser options, and other usage details of Xyce. This document is not intended to be a tutorial. Users who are new to circuit simulation are better served by the Xyce Users' Guide.

Keiter, Eric R.; Hutchinson, Scott A.; Hoekstra, Robert J.; Russo, Thomas V.

This document is intended to contain a detailed description of the mathematical formulation of Xyce, a massively parallel SPICE-style circuit simulator developed at Sandia National Laboratories. The target audience of this document are people in the role of 'service provider'. An example of such a person would be a linear solver expert who is spending a small fraction of his time developing solver algorithms for Xyce. Such a person probably is not an expert in circuit simulation, and would benefit from an description of the equations solved by Xyce. In this document, modified nodal analysis (MNA) is described in detail, with a number of examples. Issues that are unique to circuit simulation, such as voltage limiting, are also described in detail.

Keiter, Eric R.; Pawlowski, Roger P.; Hoekstra, Robert J.; Hutchinson, Scott A.; Russo, Thomas V.; Coffey, Todd S.; Kolda, Tamara G.

Abstract not provided.

Keiter, Eric R.; Pawlowski, Roger P.; Hutchinson, Scott A.; Hoekstra, Robert J.; Russo, Thomas V.; Rankin, Eric R.

Abstract not provided.

Keiter, Eric R.; Hutchinson, Scott A.; Hoekstra, Robert J.; Russo, Thomas V.; Rankin, Eric R.

Coupling between transient simulation codes of different fidelity can often be performed at the nonlinear solver level, if the time scales of the two codes are similar. A good example is electrical mixed-mode simulation, in which an analog circuit simulator is coupled to a PDE-based semiconductor device simulator. Semiconductor simulation problems, such as single-event upset (SEU), often require the fidelity of a mesh-based device simulator but are only meaningful when dynamically coupled with an external circuit. For such problems a mixed-level simulator is desirable, but the two types of simulation generally have different (somewhat conflicting) numerical requirements. To address these considerations, we have investigated variations of the two-level Newton algorithm, which preserves tight coupling between the circuit and the PDE device, while optimizing the numerics for both. The research was done within Xyce, a massively parallel electronic simulator under development at Sandia National Laboratories.

Hoekstra, Robert J.

Abstract not provided.

Heroux, Michael A.; Kolda, Tamara G.; Long, Kevin R.; Hoekstra, Robert J.; Pawlowski, Roger P.; Phipps, Eric T.; Salinger, Andrew G.; Williams, Alan B.; Heroux, Michael A.; Hu, Jonathan J.; Lehoucq, Richard B.; Thornquist, Heidi K.; Tuminaro, Raymond S.; Willenbring, James M.; Bartlett, Roscoe B.; Howle, Victoria E.

The Trilinos Project is an effort to facilitate the design, development, integration and ongoing support of mathematical software libraries. In particular, our goal is to develop parallel solver algorithms and libraries within an object-oriented software framework for the solution of large-scale, complex multi-physics engineering and scientific applications. Our emphasis is on developing robust, scalable algorithms in a software framework, using abstract interfaces for flexible interoperability of components while providing a full-featured set of concrete classes that implement all abstract interfaces. Trilinos uses a two-level software structure designed around collections of packages. A Trilinos package is an integral unit usually developed by a small team of experts in a particular algorithms area such as algebraic preconditioners, nonlinear solvers, etc. Packages exist underneath the Trilinos top level, which provides a common look-and-feel, including configuration, documentation, licensing, and bug-tracking. Trilinos packages are primarily written in C++, but provide some C and Fortran user interface support. We provide an open architecture that allows easy integration with other solver packages and we deliver our software to the outside community via the Gnu Lesser General Public License (LGPL). This report provides an overview of Trilinos, discussing the objectives, history, current development and future plans of the project.

Keiter, Eric R.; Keiter, Eric R.; Hutchinson, Scott A.; Hoekstra, Robert J.; Rankin, Eric R.; Russo, Thomas V.; Waters, Lon J.

Circuit simulation tools (e.g., SPICE) have become invaluable in the development and design of electronic circuits. Similarly, device-scale simulation tools (e.g., DaVinci) are commonly used in the design of individual semiconductor components. Some problems, such as single-event upset (SEU), require the fidelity of a mesh-based device simulator but are only meaningful when dynamically coupled with an external circuit. For such problems a mixed-level simulator is desirable, but the two types of simulation generally have different (sometimes conflicting) numerical requirements. To address these considerations, we have investigated variations of the two-level Newton algorithm, which preserves tight coupling between the circuit and the partial differential equations (PDE) device, while optimizing the numerics for both.

Hutchinson, Scott A.; Keiter, Eric R.; Hoekstra, Robert J.; Waters, Lon J.; Russo, Thomas V.; Rankin, Eric R.; Wix, Steven D.

This manual describes the use of the Xyce Parallel Electronic Simulator code for simulating electrical circuits at a variety of abstraction levels. The Xyce Parallel Electronic Simulator has been written to support,in a rigorous manner, the simulation needs of the Sandia National Laboratories electrical designers. As such, the development has focused on improving the capability over the current state-of-the-art in the following areas: (1) Capability to solve extremely large circuit problems by supporting large-scale parallel computing platforms (up to thousands of processors). Note that this includes support for most popular parallel and serial computers. (2) Improved performance for all numerical kernels (e.g., time integrator, nonlinear and linear solvers) through state-of-the-art algorithms and novel techniques. (3) A client-server or multi-tiered operating model wherein the numerical kernel can operate independently of the graphical user interface (GUI). (4) Object-oriented code design and implementation using modern coding-practices that ensure that the Xyce Parallel Electronic Simulator will be maintainable and extensible far into the future. The code is a parallel code in the most general sense of the phrase--a message passing parallel implementation--which allows it to run efficiently on the widest possible number of computing platforms. These include serial, shared-memory and distributed-memory parallel as well as heterogeneous platforms. Furthermore, careful attention has been paid to the specific nature of circuit-simulation problems to ensure that optimal parallel efficiency is achieved even as the number of processors grows. Another feature required by designers is the ability to add device models, many specific to the needs of Sandia, to the code. To this end, the device package in the Xyce Parallel Electronic Simulator is designed to support a variety of device model inputs. These input formats include standard analytical models, behavioral models and look-up tables. Combined with this flexible interface is an architectural design that greatly simplifies the addition of circuit models. One of the most important contribution Xyce makes to the designers at Sandia National Laboratories is in providing a platform for computational research and development aimed specifically at the needs of the Laboratory. With Xyce, Sandia now has an ''in-house''capability with which both new electrical (e.g., device model development) and algorithmic (e.g., faster time-integration methods) research and development can be performed. Furthermore, these capabilities will then be migrated to the end users.

Hutchinson, Scott A.; Keiter, Eric R.; Hoekstra, Robert J.; Watts, Herman A.; Waters, Lon J.; Schells, Regina L.; Wix, Steven D.

The Xyce{trademark} Parallel Electronic Simulator has been written to support the simulation needs of the Sandia National Laboratories electrical designers. As such, the development has focused on providing the capability to solve extremely large circuit problems by supporting large-scale parallel computing platforms (up to thousands of processors). In addition, they are providing improved performance for numerical kernels using state-of-the-art algorithms, support for modeling circuit phenomena at a variety of abstraction levels and using object-oriented and modern coding-practices that ensure the code will be maintainable and extensible far into the future. The code is a parallel code in the most general sense of the phrase--a message passing parallel implementation--which allows it to run efficiently on the widest possible number of computing platforms. These include serial, shared-memory and distributed-memory parallel as well as heterogeneous platforms. Furthermore, careful attention has been paid to the specific nature of circuit-simulation problems to ensure that optimal parallel efficiency is achieved even as the number of processors grows.