Publications

Heroux, Michael A.

Abstract not provided.

Willenbring, James M.; Heroux, Michael A.; Bernholdt, David E.

Abstract not provided.

Willenbring, James M.; Heroux, Michael A.; Bernholdt, David E.

Abstract not provided.

Rajamanickam, Sivasankaran R.; Heroux, Michael A.

Abstract not provided.

Communications in Computer and Information Science

Heroux, Michael A.; Gonsiorowski, Elsa; Gupta, Rinku; Milewicz, Reed M.; Moulton, J.D.; Watson, Gregory R.; Willenbring, James M.; Zamora, Richard J.; Raybourn, Elaine M.

Productivity and Sustainability Improvement Planning (PSIP) is a lightweight, iterative workflow that allows software development teams to identify development bottlenecks and track progress to overcome them. In this paper, we present an overview of PSIP and how it compares to other software process improvement (SPI) methodologies, and provide two case studies that describe how the use of PSIP led to successful improvements in team effectiveness and efficiency.

Heroux, Michael A.

Abstract not provided.

SIAM News

Heroux, Michael A.

Consider a standard SIAM journal article containing theoretical results. Each theorem has a proof that typically builds on previous developments. Since every theorem stems from a firm foundation, the research community can trust a result without further evidence. One could thus argue that a theorem does not require a proof because surely an author would not publish it if no proof existed to back it up. Furthermore, respectable reviewers and editors expect proofs without exception, and papers containing proof-less theorems will likely go unpublished.

Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Willenbring, James M.; Heroux, Michael A.; McInnes, Lois C.

Abstract not provided.

Heroux, Michael A.; Barba, Lorena B.; Parashar, Manish P.; Stodden, Victoria S.; Taufer, Michela T.

Reproducibility is an essential ingredient of the scientific enterprise. The ability to reproduce results builds trust that we can rely on the results as foundations for future scientific exploration. Presently, the fields of computational and computing sciences provide two opposing definitions of reproducible and replicable. In computational sciences, reproducible research means authors provide all necessary data and computer codes to run analyses again, so others can re-obtain the results (J. Claerbout et al., 1992). The concept was adopted and extended by several communities, where it was distinguished from replication: collecting new data to address the same question, and arriving at consistent findings (Peng et al. 2006). The Association of Computing Machinery (ACM), representing computer science and industry professionals, recently established a reproducibility initiative, adopting essentially opposite definitions. The purpose of this report is to raise awareness of the opposite definitions and propose a path to a compatible taxonomy.

Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Moulton, David M.; Raybourn, Elaine M.; McInnes, Lois C.; Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

SIAM Journal on Scientific Computing

Gamell, Marc G.; Teranishi, Keita T.; Kolla, Hemanth K.; Mayo, Jackson M.; Heroux, Michael A.; Chen, Jacqueline H.; Parashar, Manish P.

In order to achieve exascale systems, application resilience needs to be addressed. Some programming models, such as task-DAG (directed acyclic graphs) architectures, currently embed resilience features whereas traditional SPMD (single program, multiple data) and message-passing models do not. Since a large part of the community's code base follows the latter models, it is still required to take advantage of application characteristics to minimize the overheads of fault tolerance. To that end, this paper explores how recovering from hard process/node failures in a local manner is a natural approach for certain applications to obtain resilience at lower costs in faulty environments. In particular, this paper targets enabling online, semitransparent local recovery for stencil computations on current leadership-class systems as well as presents programming support and scalable runtime mechanisms. Also described and demonstrated in this paper is the effect of failure masking, which allows the effective reduction of impact on total time to solution due to multiple failures. Furthermore, we discuss, implement, and evaluate ghost region expansion and cell-to-rank remapping to increase the probability of failure masking. To conclude, this paper shows the integration of all aforementioned mechanisms with the S3D combustion simulation through an experimental demonstration (using the Titan system) of the ability to tolerate high failure rates (i.e., node failures every five seconds) with low overhead while sustaining performance at large scales. In addition, this demonstration also displays the failure masking probability increase resulting from the combination of both ghost region expansion and cell-to-rank remapping.

IEEE Transactions on Parallel and Distributed Systems

Gamell, Marc; Teranishi, Keita T.; Mayo, Jackson M.; Kolla, Hemanth K.; Heroux, Michael A.; Chen, Jacqueline H.; Parashar, Manish

Obtaining multi-process hard failure resilience at the application level is a key challenge that must be overcome before the promise of exascale can be fully realized. Previous work has shown that online global recovery can dramatically reduce the overhead of failures when compared to the more traditional approach of terminating the job and restarting it from the last stored checkpoint. If online recovery is performed in a local manner further scalability is enabled, not only due to the intrinsic lower costs of recovering locally, but also due to derived effects when using some application types. In this paper we model one such effect, namely multiple failure masking, that manifests when running Stencil parallel computations on an environment when failures are recovered locally. First, the delay propagation shape of one or multiple failures recovered locally is modeled to enable several analyses of the probability of different levels of failure masking under certain Stencil application behaviors. Our results indicate that failure masking is an extremely desirable effect at scale which manifestation is more evident and beneficial as the machine size or the failure rate increase.

Klinvex, Alicia M.; Teranishi, Keita T.; Curfman McInnes, Lois C.; Heroux, Michael A.

Abstract not provided.

Teranishi, Keita T.; Van Der Wijngaart, Rob F.; Gamell, Marc G.; Valenzuela, Eric V.; Heroux, Michael A.

Abstract not provided.

Willenbring, James M.; Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Supercomputing Frontiers and Innovations

Bartlett, Roscoe B.; Demeshko, Irina; Gamblin, Todd; Hammond, Glenn E.; Heroux, Michael A.; Johnson, Jeffrey; Klinvex, Alicia M.; Li, Xiaoye; McInnes, Lois C.; Moulton, J.D.; Osei-Kuffuor, Daniel; Sarich, Jason; Smith, Barry; Willenbring, James M.; Yang, Ulrike M.

Extreme-scale computational science increasingly demands multiscale and multiphysics formulations. Combining software developed by independent groups is imperative: no single team has resources for all predictive science and decision support capabilities. Scientific libraries provide high-quality, reusable software components for constructing applications with improved robustness and portability. However, without coordination, many libraries cannot be easily composed. Namespace collisions, inconsistent arguments, lack of third-party software versioning, and additional difficulties make composition costly. The Extreme-scale Scientific Software Development Kit (xSDK) defines community policies to improve code quality and compatibility across independently developed packages (hypre, PETSc, SuperLU, Trilinos, and Alquimia) and provides a foundation for addressing broader issues in software interoperability, performance portability, and sustainability. The xSDK provides turnkey installation of member software and seamless combination of aggregate capabilities, and it marks first steps toward extreme-scale scientific software ecosystems from which future applications can be composed rapidly with assured quality and scalability.

Van Der Wijngaart, Rob F.; Gamell, Marc G.; Teranishi, Keita T.; Valenzuela, Eric V.; Heroux, Michael A.; Parashar, Manish P.

Abstract not provided.

Van Der Wijngaart, Rob F.; Gamell, Marc G.; Teranishi, Keita T.; Valenzuela, Eric V.; Heroux, Michael A.; Parashaar, Manish P.

Abstract not provided.

Proceedings of the International Conference on Parallel Processing Workshops

Gamell, Marc; Katz, Daniel S.; Teranishi, Keita T.; Heroux, Michael A.; Van Der Wijngaart, Rob F.; Mattson, Timothy G.; Parashar, Manish

Exascale systems promise the potential for computation atunprecedented scales and resolutions, but achieving exascale by theend of this decade presents significant challenges. A key challenge isdue to the very large number of cores and components and the resultingmean time between failures (MTBF) in the order of hours orminutes. Since the typical run times of target scientific applicationsare longer than this MTBF, fault tolerance techniques will beessential. An important class of failures that must be addressed isprocess or node failures. While checkpoint/restart (C/R) is currentlythe most widely accepted technique for addressing processor failures, coordinated, stable-storage-based global C/R might be unfeasible atexascale when the time to checkpoint exceeds the expected MTBF. This paper explores transparent recovery via implicitly coordinated, diskless, application-driven checkpointing as a way to tolerateprocess failures in MPI applications at exascale. The discussedapproach leverages User Level Failure Mitigation (ULFM), which isbeing proposed as an MPI extension to allow applications to createpolicies for tolerating process failures. Specifically, this paper demonstrates how different implementations ofapplication-driven in-memory checkpoint storage and recovery comparein terms of performance and scalability. We also experimentally evaluate the effectiveness and scalability ofthe Fenix online global recovery framework on a production system-the Titan Cray XK7 at ORNL-and demonstrate the ability of Fenix totolerate dynamically injected failures using the execution of fourbenchmarks and mini-applications with different behaviors.

Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

Teranishi, Keita T.; Gamell, Marc G.; Van Der Vijngarrt, Rob V.; Heroux, Michael A.; Parashar, Manish P.

Abstract not provided.

Leland, Robert; Rajan, Mahesh R.; Heroux, Michael A.

Our first purpose here is to offer to a general technical and policy audience a perspective on whether the supercomputing community should focus on improving the efficiency of supercomputing systems and their use rather than on building larger and ostensibly more capable systems that are used at low efficiency. After first summarizing our content and defining some necessary terms, we give a concise answer to this question. We then set this in context by characterizing performance of current supercomputing systems on a variety of benchmark problems and actual problems drawn from workloads in the national security, industrial, and scientific context. Along the way we answer some related questions, identify some important technological trends, and offer a perspective on the significance of these trends. Our second purpose is to give a reasonably broad and transparent overview of the related issue space and thereby to better equip the reader to evaluate commentary and controversy concerning supercomputing performance. For example, questions repeatedly arise concerning the Linpack benchmark and its predictive power, so we consider this in moderate depth as an example. We also characterize benchmark and application performance for scientific and engineering use of supercomputers and offer some guidance on how to think about these. Examples here are drawn from traditional scientific computing. Other problem domains, for example, data analytics, have different performance characteristics that are better captured by different benchmark problems or applications, but the story in those domains is similar in character and leads to similar conclusions with regard to the motivating question. For more on this topic, see Large-Scale Data Analytics and Its Relationship to Simulation. 1 Director, Computing Research Center, Sandia National Laboratories 2 Distinguished Member of the Technical Staff, Sandia National Laboratories 3 Distinguished Member of the Technical Staff, Sandia National Laboratories 4 Distinguished Member of the Technical Staff , Sandia National Laboratories

Teranishi, Keita T.; Gamell, Marc G.; Van Der Wijngaart, Rob F.; Parashar, Manish P.; Heroux, Michael A.

Abstract not provided.

Teranishi, Keita T.; Gmaell, Marc G.; Van Der Wijngarrt, Rob V.; Parashar, Manish P.; Heroux, Michael A.

Abstract not provided.

CEUR Workshop Proceedings

Heroux, Michael A.

Computational Science and Engineering (CSE) software can benefit substantially from an explicit focus on quality improvement. This is especially true as we face increased demands in both modeling and software complexities. At the same time, just desiring improved quality is not sufficient. We must work with the entities that provide CSE research teams with publication venues, funding, and professional recognition in order to increase incentives for improved software quality. In fact, software quality is precisely calibrated to the expectations, explicit and implicit, set by these entities. We will see broad improvements in sustainability and productivity only when publishers, funding agencies and employers raise their expectations for software quality. CSE software community leaders, those who are in a position to inform and influence these entities, have a unique opportunity to broadly and positively impact software quality by working to establish incentives that will spur creative and novel approaches to improve developer productivity and software sustainability.

Rajamanickam, Sivasankaran R.; Yamazaki, I.Y.; Boman, Erik G.; Prokopenko, Andrey V.; Heroux, Michael A.; Dongarra, J.D.

Abstract not provided.

Heroux, Michael A.

Abstract not provided.

International Conference for High Performance Computing, Networking, Storage and Analysis, SC

Gamell, Marc; Teranishi, Keita T.; Heroux, Michael A.; Mayo, Jackson M.; Kolla, Hemanth K.; Chen, Jacqueline H.; Parashar, Manish

Application resilience is a key challenge that has to be addressed to realize the exascale vision. Online recovery, even when it involves all processes, can dramatically reduce the overhead of failures as compared to the more traditional approach where the job is terminated and restarted from the last checkpoint. In this paper we explore how local recovery can be used for certain classes of applications to further reduce overheads due to resilience. Specifically we develop programming support and scalable runtime mechanisms to enable online and transparent local recovery for stencil-based parallel applications on current leadership class systems. We also show how multiple independent failures can be masked to effectively reduce the impact on the total time to solution. We integrate these mechanisms with the S3D combustion simulation, and experimentally demonstrate (using the Titan Cray-XK7 system at ORNL) the ability to tolerate high failure rates (i.e., node failures every 5 seconds) with low overhead while sustaining performance, at scales up to 262144 cores.

Teranishi, Keita T.; Heroux, Michael A.

Abstract not provided.

Demeshko, Irina D.; Heroux, Michael A.; Salinger, Andrew G.

Abstract not provided.

Rajamanickam, Sivasankaran R.; Yamazaki, Ichitaro Y.; Boman, Erik G.; Hoemmen, Mark F.; Heroux, Michael A.; Tomov, Stan T.; Dongarra, Jack D.

Abstract not provided.

International Journal of High Performance Computing Applications

Heroux, Michael A.; Dongarra, Jack D.; Luszczek, Piotr L.

Abstract not provided.

Gamell Balmana, Marc G.; Teranishi, Keita T.; Heroux, Michael A.; Mayo, Jackson M.; Kolla, Hemanth K.; Chen, Jacqueline H.; Parashar, Manish P.

Abstract not provided.

Concurrency and Computation. Practice and Experience

Lin, Paul L.; Heroux, Michael A.; Williams, Alan B.; Barrett, Richard F.

The performance of a large-scale, production-quality science and engineering application (‘app’) is often dominated by a small subset of the code. Even within that subset, computational and data access patterns are often repeated, so that an even smaller portion can represent the performance-impacting features. If application developers, parallel computing experts, and computer architects can together identify this representative subset and then develop a small mini-application (‘miniapp’) that can capture these primary performance characteristics, then this miniapp can be used to both improve the performance of the app as well as provide a tool for co-design for the high-performance computing community. However, a critical question is whether a miniapp can effectively capture key performance behavior of an app. This study provides a comparison of an implicit finite element semiconductor device modeling app on unstructured meshes with an implicit finite element miniapp on unstructured meshes. The goal is to assess whether the miniapp is predictive of the performance of the app. Finally, single compute node performance will be compared, as well as scaling up to 16,000 cores. Results indicate that the miniapp can be reasonably predictive of the performance characteristics of the app for a single iteration of the solver on a single compute node.

HPDC 2015 - Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing

Gamell, Marc; Teranishi, Keita T.; Heroux, Michael A.; Mayo, Jackson M.; Kolla, Hemanth K.; Chen, Jacqueline H.; Parashar, Manish

Application resilience is a key challenge that must be ad-dressed in order to realize the exascale vision. Previous work has shown that online recovery, even when done in a global manner (i.e., involving all processes), can dramatically re-duce the overhead of failures when compared to the more traditional approach of terminating the job and restarting it from the last stored checkpoint. In this paper we suggest going one step further, and explore how local recovery can be used for certain classes of applications to reduce the over-heads due to failures. Specifically we study the feasibility of local recovery for stencil-based parallel applications and we show how multiple independent failures can be masked to effectively reduce the impact on the total time to solution.

Demeshko, Irina D.; Salinger, Andrew G.; Heroux, Michael A.

Abstract not provided.