Publications

Geronimo Anderson, Sean I.; Teranishi, Keita T.; Dunlavy, Daniel D.; Choi, Jee C.

Abstract not provided.

Geronimo Anderson, Sean I.; Teranishi, Keita T.; Dunlavy, Daniel D.; Choi, Jee C.

Abstract not provided.

Giem, Elisabeth A.; Poliakoff, David Z.; Teranishi, Keita T.; Hollman, Daisy H.

Abstract not provided.

Morales, Nicolas M.; Teranishi, Keita T.; Nicolae, Bogdan N.; Trott, Christian R.; Capello, Franck C.

Abstract not provided.

2020 IEEE High Performance Extreme Computing Conference, HPEC 2020

Teranishi, Keita T.; Dunlavy, Daniel D.; Myers, Jeremy M.; Barrett, Richard F.

Canonical Polyadic tensor decomposition using alternate Poisson regression (CP-APR) is an effective analysis tool for large sparse count datasets. One of the variants using projected damped Newton optimization for row subproblems (PDNR) offers quadratic convergence and is amenable to parallelization. Despite its potential effectiveness, PDNR performance on modern high performance computing (HPC) systems is not well understood. To remedy this, we have developed a parallel implementation of PDNR using Kokkos, a performance portable parallel programming framework supporting efficient runtime of a single code base on multiple HPC systems. We demonstrate that the performance of parallel PDNR can be poor if load imbalance associated with the irregular distribution of nonzero entries in the tensor data is not addressed. Preliminary results using tensors from the FROSTT data set indicate that using multiple kernels to address this imbalance when solving the PDNR row subproblems in parallel can improve performance, with up to 80% speedup on CPUs and 10-fold speedup on NVIDIA GPUs.

2020 IEEE High Performance Extreme Computing Conference, HPEC 2020

Myers, Jeremy M.; Dunlavy, Daniel D.; Teranishi, Keita T.; Hollman, David S.

Tensor decomposition models play an increasingly important role in modern data science applications. One problem of particular interest is fitting a low-rank Canonical Polyadic (CP) tensor decomposition model when the tensor has sparse structure and the tensor elements are nonnegative count data. SparTen is a high-performance C++ library which computes a low-rank decomposition using different solvers: a first-order quasi-Newton or a second-order damped Newton method, along with the appropriate choice of runtime parameters. Since default parameters in SparTen are tuned to experimental results in prior published work on a single real-world dataset conducted using MATLAB implementations of these methods, it remains unclear if the parameter defaults in SparTen are appropriate for general tensor data. Furthermore, it is unknown how sensitive algorithm convergence is to changes in the input parameter values. This report addresses these unresolved issues with large-scale experimentation on three benchmark tensor data sets. Experiments were conducted on several different CPU architectures and replicated with many initial states to establish generalized profiles of algorithm convergence behavior.

Myers, Jeremy M.; Dunlavy, Daniel D.; Teranishi, Keita T.; Hollman, David S.

Abstract not provided.

Teranishi, Keita T.; Dunlavy, Daniel D.; Myers, Jeremy M.; Barrett, Richard F.

Abstract not provided.

Teranishi, Keita T.; Morales, Nicolas M.; Miles, Jeffery S.; Mould, Carson M.; Whitlock, Matthew J.

Abstract not provided.

Morales, Nicolas M.; Miles, Jeffery S.; Mould, Carson M.; Teranishi, Keita T.; Nicolae, Bogdan N.

Abstract not provided.

Teranishi, Keita T.; Hollman, David S.; Myers, Jeremy M.; Dunlavy, Daniel D.

Abstract not provided.

Miles, Jeffery S.; Teranishi, Keita T.; Morales, Nicolas M.; Mould, Carson M.

Abstract not provided.

Miles, Jeffery S.; Teranishi, Keita T.; Morales, Nicolas M.; Mould, Carson M.

Abstract not provided.

Teranishi, Keita T.; Hollman, David S.; Barrett, Richard F.; Myers, Jeremy M.; Dunlavy, Daniel D.

Abstract not provided.

Miles, Jeffery S.; Morales, Nicolas M.; Teranishi, Keita T.

Abstract not provided.

Teranishi, Keita T.; Hollman, David S.; Dunlavy, Daniel D.; Barrett, Richard F.

Abstract not provided.

Miles, Jeffery S.; Morales, Nicolas M.; Teranishi, Keita T.; Trott, Christian R.

Abstract not provided.

Teranishi, Keita T.; Dunlavy, Daniel D.; Barrett, Richard F.; Kolda, Tamara G.; Forster, Christopher F.

Abstract not provided.

Teranishi, Keita T.; Barrett, Richard F.; Forster, Christopher F.; Dunlavy, Daniel D.; Kolda, Tamara G.

Abstract not provided.

SIAM Journal on Scientific Computing

Gamell, Marc G.; Teranishi, Keita T.; Kolla, Hemanth K.; Mayo, Jackson M.; Heroux, Michael A.; Chen, Jacqueline H.; Parashar, Manish P.

In order to achieve exascale systems, application resilience needs to be addressed. Some programming models, such as task-DAG (directed acyclic graphs) architectures, currently embed resilience features whereas traditional SPMD (single program, multiple data) and message-passing models do not. Since a large part of the community's code base follows the latter models, it is still required to take advantage of application characteristics to minimize the overheads of fault tolerance. To that end, this paper explores how recovering from hard process/node failures in a local manner is a natural approach for certain applications to obtain resilience at lower costs in faulty environments. In particular, this paper targets enabling online, semitransparent local recovery for stencil computations on current leadership-class systems as well as presents programming support and scalable runtime mechanisms. Also described and demonstrated in this paper is the effect of failure masking, which allows the effective reduction of impact on total time to solution due to multiple failures. Furthermore, we discuss, implement, and evaluate ghost region expansion and cell-to-rank remapping to increase the probability of failure masking. To conclude, this paper shows the integration of all aforementioned mechanisms with the S3D combustion simulation through an experimental demonstration (using the Titan system) of the ability to tolerate high failure rates (i.e., node failures every five seconds) with low overhead while sustaining performance at large scales. In addition, this demonstration also displays the failure masking probability increase resulting from the combination of both ghost region expansion and cell-to-rank remapping.

IEEE Transactions on Parallel and Distributed Systems

Gamell, Marc; Teranishi, Keita T.; Mayo, Jackson M.; Kolla, Hemanth K.; Heroux, Michael A.; Chen, Jacqueline H.; Parashar, Manish

Obtaining multi-process hard failure resilience at the application level is a key challenge that must be overcome before the promise of exascale can be fully realized. Previous work has shown that online global recovery can dramatically reduce the overhead of failures when compared to the more traditional approach of terminating the job and restarting it from the last stored checkpoint. If online recovery is performed in a local manner further scalability is enabled, not only due to the intrinsic lower costs of recovering locally, but also due to derived effects when using some application types. In this paper we model one such effect, namely multiple failure masking, that manifests when running Stencil parallel computations on an environment when failures are recovered locally. First, the delay propagation shape of one or multiple failures recovered locally is modeled to enable several analyses of the probability of different levels of failure masking under certain Stencil application behaviors. Our results indicate that failure masking is an extremely desirable effect at scale which manifestation is more evident and beneficial as the machine size or the failure rate increase.

Teranishi, Keita T.; Forster, Christopher J.; Dunlavy, Daniel D.; Kolda, Tamara G.

Abstract not provided.

Klinvex, Alicia M.; Teranishi, Keita T.; Curfman McInnes, Lois C.; Heroux, Michael A.

Abstract not provided.

Rizzi, Francesco N.; Phipps, Eric T.; Hollman, David S.; Lifflander, Jonathan; Wilke, Jeremiah J.; Markosyan, Aram H.; Kolla, Hemanth K.; Slattengren, Nicole S.; Teranishi, Keita T.; Stewart, James R.; Clay, Robert L.; Bennett, Janine C.

Abstract not provided.

Teranishi, Keita T.; Forster, Christopher J.; Dunlavy, Daniel D.; Kolda, Tamara G.

Abstract not provided.

Gammel, Marc G.; Teranishi, Keita T.; Knight, Samuel K.; Sjaardema, Gregory D.; Kolla, Hemanth K.; Wilke, Jason W.; Slattengren, Nicole S.; Ferreira, Kurt B.; Bennett, Janine C.; Jain, Nikhil J.; Kale, Laxmikant K.

Abstract not provided.

Teranishi, Keita T.; Van Der Wijngaart, Rob F.; Gamell, Marc G.; Valenzuela, Eric V.; Heroux, Michael A.

Abstract not provided.

Van Der Wijngaart, Rob F.; Gamell, Marc G.; Teranishi, Keita T.; Valenzuela, Eric V.; Heroux, Michael A.; Parashar, Manish P.

Abstract not provided.

Van Der Wijngaart, Rob F.; Gamell, Marc G.; Teranishi, Keita T.; Valenzuela, Eric V.; Heroux, Michael A.; Parashaar, Manish P.

Abstract not provided.

Proceedings of the International Conference on Parallel Processing Workshops

Gamell, Marc; Katz, Daniel S.; Teranishi, Keita T.; Heroux, Michael A.; Van Der Wijngaart, Rob F.; Mattson, Timothy G.; Parashar, Manish

Exascale systems promise the potential for computation atunprecedented scales and resolutions, but achieving exascale by theend of this decade presents significant challenges. A key challenge isdue to the very large number of cores and components and the resultingmean time between failures (MTBF) in the order of hours orminutes. Since the typical run times of target scientific applicationsare longer than this MTBF, fault tolerance techniques will beessential. An important class of failures that must be addressed isprocess or node failures. While checkpoint/restart (C/R) is currentlythe most widely accepted technique for addressing processor failures, coordinated, stable-storage-based global C/R might be unfeasible atexascale when the time to checkpoint exceeds the expected MTBF. This paper explores transparent recovery via implicitly coordinated, diskless, application-driven checkpointing as a way to tolerateprocess failures in MPI applications at exascale. The discussedapproach leverages User Level Failure Mitigation (ULFM), which isbeing proposed as an MPI extension to allow applications to createpolicies for tolerating process failures. Specifically, this paper demonstrates how different implementations ofapplication-driven in-memory checkpoint storage and recovery comparein terms of performance and scalability. We also experimentally evaluate the effectiveness and scalability ofthe Fenix online global recovery framework on a production system-the Titan Cray XK7 at ORNL-and demonstrate the ability of Fenix totolerate dynamically injected failures using the execution of fourbenchmarks and mini-applications with different behaviors.

Teranishi, Keita T.; Gamell, Marc G.; Van Der Vijngarrt, Rob V.; Heroux, Michael A.; Parashar, Manish P.

Abstract not provided.

Teranishi, Keita T.; Gamell, Marc G.; Van Der Wijngaart, Rob F.; Parashar, Manish P.; Heroux, Michael A.

Abstract not provided.

Teranishi, Keita T.; Gmaell, Marc G.; Van Der Wijngarrt, Rob V.; Parashar, Manish P.; Heroux, Michael A.

Abstract not provided.

International Conference for High Performance Computing, Networking, Storage and Analysis, SC

Gamell, Marc; Teranishi, Keita T.; Heroux, Michael A.; Mayo, Jackson M.; Kolla, Hemanth K.; Chen, Jacqueline H.; Parashar, Manish

Application resilience is a key challenge that has to be addressed to realize the exascale vision. Online recovery, even when it involves all processes, can dramatically reduce the overhead of failures as compared to the more traditional approach where the job is terminated and restarted from the last checkpoint. In this paper we explore how local recovery can be used for certain classes of applications to further reduce overheads due to resilience. Specifically we develop programming support and scalable runtime mechanisms to enable online and transparent local recovery for stencil-based parallel applications on current leadership class systems. We also show how multiple independent failures can be masked to effectively reduce the impact on the total time to solution. We integrate these mechanisms with the S3D combustion simulation, and experimentally demonstrate (using the Titan Cray-XK7 system at ORNL) the ability to tolerate high failure rates (i.e., node failures every 5 seconds) with low overhead while sustaining performance, at scales up to 262144 cores.

Teranishi, Keita T.; Heroux, Michael A.

Abstract not provided.

Baker, Gavin M.; Bettencourt, Matthew T.; Bova, S.W.; franko, ken f.; Gamell, Marc G.; Grant, Ryan E.; Hammond, Simon D.; Hollman, David S.; Knight, Samuel K.; Kolla, Hemanth K.; Lin, Paul L.; Olivier, Stephen O.; Sjaardema, Gregory D.; Slattengren, Nicole L.; Teranishi, Keita T.; Wilke, Jeremiah J.; Bennett, Janine C.; Clay, Robert L.; kale, laxkimant k.; Jain, Nikhil J.; Mikida, Eric M.; Aiken, Alex A.; Bauer, Michael B.; Lee, Wonchan L.; Slaughter, Elliott S.; Treichler, Sean T.; Berzins, Martin B.; Harman, Todd H.; humphreys, alan h.; schmidt, john s.; sunderland, dan s.; Mccormick, Pat M.; gutierrez, samuel g.; shulz, martin s.; Gamblin, Todd G.; Bremer, Peer-Timo B.

Abstract not provided.

Baker, Gavin M.; Bettencourt, Matthew T.; Bova, S.W.; franko, ken f.; Gamell, Marc G.; Grant, Ryan E.; Hammond, Simon D.; Hollman, David S.; Knight, Samuel K.; Kolla, Hemanth K.; Lin, Paul L.; Olivier, Stephen O.; Sjaardema, Gregory D.; Slattengren, Nicole L.; Teranishi, Keita T.; Wilke, Jeremiah J.; Bennett, Janine C.; Clay, Robert L.; kale, laxkimant k.; Jain, Nikhil J.; Mikida, Eric M.; Aiken, Alex A.; Bauer, Michael B.; Lee, Wonchan L.; Slaughter, Elliott S.; Treichler, Sean T.; Berzins, Martin B.; Harman, Todd H.; humphreys, alan h.; schmidt, john s.; sunderland, dan s.; Mccormick, Pat M.; gutierrez, samuel g.; shulz, martin s.; Gamblin, Todd G.; Bremer, Peer-Timo B.

This report provides in-depth information and analysis to help create a technical road map for developing next-generation programming models and runtime systems that support Advanced Simulation and Computing (ASC) work- load requirements. The focus herein is on asynchronous many-task (AMT) model and runtime systems, which are of great interest in the context of "Oriascale7 computing, as they hold the promise to address key issues associated with future extreme-scale computer architectures. This report includes a thorough qualitative and quantitative examination of three best-of-class AIM] runtime systems – Charm-++, Legion, and Uintah, all of which are in use as part of the Centers. The studies focus on each of the runtimes' programmability, performance, and mutability. Through the experiments and analysis presented, several overarching Predictive Science Academic Alliance Program II (PSAAP-II) Asc findings emerge. From a performance perspective, AIV runtimes show tremendous potential for addressing extreme- scale challenges. Empirical studies show an AM runtime can mitigate performance heterogeneity inherent to the machine itself and that Message Passing Interface (MP1) and AM11runtimes perform comparably under balanced conditions. From a programmability and mutability perspective however, none of the runtimes in this study are currently ready for use in developing production-ready Sandia ASC applications. The report concludes by recommending a co- design path forward, wherein application, programming model, and runtime system developers work together to define requirements and solutions. Such a requirements-driven co-design approach benefits the community as a whole, with widespread community engagement mitigating risk for both application developers developers. and high-performance computing runtime systein

Wilke, Jeremiah J.; Bettencourt, Matthew T.; Bova, S.W.; franko, ken f.; Gamell, Marc G.; Grant, Ryan E.; Hammond, Simon D.; Hollman, David S.; Knight, Samuel K.; Kolla, Hemanth K.; Lin, Paul L.; Olivier, Stephen L.; Sjaardema, Gregory D.; Slattengren, Nicole S.; Teranishi, Keita T.; Bennett, Janine C.; Clay, Robert L.

Abstract not provided.

Gamell Balmana, Marc G.; Teranishi, Keita T.; Heroux, Michael A.; Mayo, Jackson M.; Kolla, Hemanth K.; Chen, Jacqueline H.; Parashar, Manish P.

Abstract not provided.

HPDC 2015 - Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing

Gamell, Marc; Teranishi, Keita T.; Heroux, Michael A.; Mayo, Jackson M.; Kolla, Hemanth K.; Chen, Jacqueline H.; Parashar, Manish

Application resilience is a key challenge that must be ad-dressed in order to realize the exascale vision. Previous work has shown that online recovery, even when done in a global manner (i.e., involving all processes), can dramatically re-duce the overhead of failures when compared to the more traditional approach of terminating the job and restarting it from the last stored checkpoint. In this paper we suggest going one step further, and explore how local recovery can be used for certain classes of applications to reduce the over-heads due to failures. Specifically we study the feasibility of local recovery for stencil-based parallel applications and we show how multiple independent failures can be masked to effectively reduce the impact on the total time to solution.

Bennett, Janine C.; Wilke, Jeremiah J.; Slattengren, Nicole S.; Teranishi, Keita T.; Lin, Paul L.; Sjaardema, Gregory D.; Kolla, Hemanth K.; Hollman, David S.; Knight, Samuel K.; franko, ken f.; Clay, Robert L.

Abstract not provided.

Gamell Balmana, Marc G.; Teranishi, Keita T.; Heroux, Michael A.; Mayo, Jackson M.; Kolla, Hemanth K.; Chen, Jacqueline H.; Parashar, Manish P.

Abstract not provided.

Bennett, Janine C.; Wilke, Jeremiah J.; Slattengren, Nicole S.; Teranishi, Keita T.; Lin, Paul L.; Sjaardema, Gregory D.; Kolla, Hemanth K.

Abstract not provided.

Bennett, Janine C.; Wilke, Jeremiah J.; Slattengren, Nicole S.; Teranishi, Keita T.; Lin, Paul L.; Sjaardema, Gregory D.; Kolla, Hemanth K.; Knight, Samuel K.

Abstract not provided.

Bennett, Janine C.; Wilke, Jeremiah J.; Slattengren, Nicole S.; Teranishi, Keita T.; Kolla, Hemanth K.; Lin, Paul L.; Sjaardema, Gregory D.

Abstract not provided.

Hollman, David S.; Hollman, David S.; Bennett, Janine C.; Bennett, Janine C.; Wilke, Jeremiah J.; Wilke, Jeremiah J.; Kolla, Hemanth K.; Kolla, Hemanth K.; Lin, Paul L.; Lin, Paul L.; Slattengren, Nicole S.; Slattengren, Nicole S.; Teranishi, Keita T.; Teranishi, Keita T.; franko, ken f.; franko, ken f.; Jain, Nikhil J.; Jain, Nikhil J.; Mikida, Eric M.; Mikida, Eric M.

Abstract not provided.

Bennett, Janine C.; Wilke, Jeremiah J.; Slattengren, Nicole S.; Teranishi, Keita T.; Franko, Kenneth J.; Sjaardema, Gregory D.; Lin, Paul L.; Kolla, Hemanth K.

Abstract not provided.

Procedia Computer Science

Chien, A.; Balaji, P.; Beckman, P.; Dun, N.; Fang, A.; Fujita, H.; Iskra, K.; Rubenstein, Z.; Zheng, Z.; Schreiber, R.; Hammond, J.; Dinan, J.; Laguna, I.; Richards, D.; Dubey, A.; Van Straalen, B.; Hoemmen, M.; Heroux, Michael A.; Teranishi, Keita T.; Siegel, A.

Exascale studies project reliability challenges for future high-performance computing (HPC) systems. We propose the Global View Resilience (GVR) system, a library that enables applications to add resilience in a portable, application-controlled fashion using versioned distributed arrays. We describe GVR's interfaces to distributed arrays, versioning, and cross-layer error recovery. Using several large applications (OpenMC, the preconditioned conjugate gradient solver PCG, ddcMD, and Chombo), we evaluate the programmer effort to add resilience. The required changes are small (<2% LOC), localized, and machine-independent, requiring no software architecture changes. We also measure the overhead of adding GVR versioning and show that generally overheads <2% are achieved. We conclude that GVR's interfaces and implementation are flexible and portable and create a gentle-slope path to tolerate growing error rates in future systems.

Gamell, Marc G.; Teranishi, Keita T.; Heroux, Michael A.; Mayo, Jackson M.; Kolla, Hemanth K.; Chen, Jacqueline H.; Parashar, Manish P.

Abstract not provided.

Franko, Kenneth J.; Sjaardema, Gregory D.; Bennett, Janine C.; Kolla, Hemanth K.; Lin, Paul L.; Teranishi, Keita T.; Wilke, Jeremiah J.

Abstract not provided.

ACM International Conference Proceeding Series

Teranishi, Keita T.; Heroux, Michael A.

The current system reaction to the loss of a single MPI process is to kill all the remaining processes and restart the application from the most recent checkpoint. This approach will become unfeasible for future extreme scale systems. We address this issue using an emerging resilient computing model called Local Failure Local Recovery (LFLR) that provides application developers with the ability to recover locally and continue application execution when a process is lost. We discuss the design of our software framework to enable the LFLR model using MPI-ULFM and demonstrate the resilient version of MiniFE that achieves a scalable recovery from process failures.

Teranishi, Keita T.; Heroux, Michael A.; Gamell Balmana, Marc G.; Parashar, Manish P.

Abstract not provided.

Teranishi, Keita T.; Gamell Balmana, Marc G.; Heroux, Michael A.; Parashar, Manish, R.

Abstract not provided.

Teranishi, Keita T.; Heroux, Michael A.

Abstract not provided.

Heroux, Michael A.; Teranishi, Keita T.

Recovery from process loss during the execution of a distributed memory parallel application is presently achieved by restarting the program, typically from a checkpoint file. Future computer system trends indicate that the size of data to checkpoint, the lack of improvement in parallel file system performance and the increase in process failure rates will lead to situations where checkpoint restart becomes infeasible. In this report we describe and prototype the use of a new application level resilient computing model that manages persistent storage of local state for each process such that, if a process fails, recovery can be performed locally without requiring access to a global checkpoint file. LFLR provides application developers with an ability to recover locally and continue application execution when a process is lost. This report discusses what features are required from the hardware, OS and runtime layers, and what approaches application developers might use in the design of future codes, including a demonstration of LFLR-enabled MiniFE code from the Matenvo mini-application suite.