Publications

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Results from the new DOE super-parameterized (SP) Energy Exascale Earth System Model (SP-E3SM) are analyzed and compared to the traditionally parameterized E3SMv1 and previous studies using SP models. SP-E3SM is unique in that it utilizes GPU hardware acceleration, CRM mean-state acceleration, and reduced radiation to dramatically increase the model throughput and allow decadal experiments at 100-km external resolution. It also differs from other SP models by using a spectral element dynamical core on a cubed sphere grid and a finer vertical grid with a higher model top. Despite these differences, SP-E3SM generally reproduces the behavior of other super-parameterized models. Tropical wave variability is improved relative to E3SM, including the emergence of a Madden-Julian Oscillation and a realistic slowdown of Moist Kelvin Waves. However, the distribution of precipitation exhibits an unrealistically large variance, and while the timing of diurnal rainfall shows modest improvements the signal is not as coherent as observations. A notable grid imprinting bias is identified in the precipitation field and attributed to a unique feedback associated with the interactions between explicit convection and the spectral element grid structure. Spurious zonal mean column water tendencies due to grid imprinting are quantified – while negligible for the conventionally parameterized E3SM, they become large with super-parameterization, approaching 10% of the physical tendencies. The implication is that finding a remedy to grid imprinting will become especially important as spectral element dynamical cores begin to be combined with explicitly resolved convection.

We derive a formulation of the nonhydrostatic equations in spherical geometry with a Lorenz staggered vertical discretization. The combination conserves a discrete energy in exact time integration when coupled with a mimetic horizontal discretization. The formulation is a version of Dubos and Tort (2014, https://doi.org/10.1175/MWR-D-14-00069.1) rewritten in terms of primitive variables. It is valid for terrain following mass or height coordinates and for both Eulerian or vertically Lagrangian discretizations. The discretization relies on an extension to Simmons and Burridge (1981, https://doi.org/10.1175/1520-0493(1981)109<0758:AEAAMC>2.0.CO;2) vertical differencing, which we show obeys a discrete derivative product rule. This product rule allows us to simplify the treatment of the vertical transport terms. Energy conservation is obtained via a term-by-term balance in the kinetic, internal, and potential energy budgets, ensuring an energy-consistent discretization up to time truncation error with no spurious sources of energy. We demonstrate convergence with respect to time truncation error in a spectral element code with a horizontal explicit vertically implicit implicit-explicit time stepping algorithm.

We present a Fourier analysis of wave propagation problems subject to a class of continuous and discontinuous discretizations using high-degree Lagrange polynomials. This allows us to obtain explicit analytical formulas for the dispersion relation and group velocity and, for the first time to our knowledge, characterize analytically the emergence of gaps in the dispersion relation at specific wavenumbers, when they exist, and compute their specific locations. Wave packets with energy at these wavenumbers will fail to propagate correctly, leading to significant numerical dispersion. We also show that the Fourier analysis generates mathematical artifacts, and we explain how to remove them through a branch selection procedure conducted by analysis of eigenvectors and associated reconstructed solutions. The higher frequency eigenmodes, named erratic in this study, are also investigated analytically and numerically.

This study provides an overview of the coupled high-resolution Version 1 of the Energy Exascale Earth System Model (E3SMv1) and documents the characteristics of a 50-year-long high-resolution control simulation with time-invariant 1950 forcings following the HighResMIP protocol. In terms of global root-mean-squared error metrics, this high-resolution simulation is generally superior to results from the low-resolution configuration of E3SMv1 (due to resolution, tuning changes, and possibly initialization procedure) and compares favorably to models in the CMIP5 ensemble. Ocean and sea ice simulation is particularly improved, due to better resolution of bathymetry, the ability to capture more variability and extremes in winds and currents, and the ability to resolve mesoscale ocean eddies. The largest improvement in this regard is an ice-free Labrador Sea, which is a major problem at low resolution. Interestingly, several features found to improve with resolution in previous studies are insensitive to resolution or even degrade in E3SMv1. Most notable in this regard are warm bias and associated stratocumulus deficiency in eastern subtropical oceans and lack of improvement in El Niño. Another major finding of this study is that resolution increase had negligible impact on climate sensitivity (measured by net feedback determined through uniform +4K prescribed sea surface temperature increase) and aerosol sensitivity. Cloud response to resolution increase consisted of very minor decrease at all levels. Large-scale patterns of precipitation bias were also relatively unaffected by grid spacing.

Abstract not provided.

Concern over Arctic methane (CH₄) emissions has increased following recent discoveries of poorly understood sources and predictions that methane emissions from known sources will grow as Arctic temperatures increase. New efforts are required to detect increases and explain sources without being confounded by the multiple sources. Methods for distinguishing different sources are critical. We conducted measurements of atmospheric methane and source tracers and performed baseline global atmospheric modeling to begin assessing the climate impact of changes in atmospheric methane. The goal of this project was to address uncertainties in Arctic methane sources and their potential impact on climate by (1) deploying newly developed trace-gas analyzers for measurements of methane, methane isotopologues, ethane, and other tracers of methane sources in the Barrow, AK, (2) characterizing methane sources using high-resolution atmospheric chemical transport models and tracer measurements, and (3) modeling Arctic climate using the state-of-the-art high- resolution Spectral Element Community Atmosphere Model (CAM-SE).

We present a new method for reducing parallel applications’ communication time by mapping their MPI tasks to processors in a way that lowers the distance messages travel and the amount of congestion in the network. Assuming geometric proximity among the tasks is a good approximation of their communication interdependence, we use a geometric partitioning algorithm to order both the tasks and the processors, assigning task parts to the corresponding processor parts. In this way, interdependent tasks are assigned to “nearby” cores in the network. We also present a number of algorithmic optimizations that exploit specific features of the network or application to further improve the quality of the mapping. We specifically address the case of sparse node allocation, where the nodes assigned to a job are not necessarily located in a contiguous block nor within close proximity to each other in the network. However, our methods generalize to contiguous allocations as well, and results are shown for both contiguous and non-contiguous allocations. We show that, for the structured finite difference mini-application MiniGhost, our mapping methods reduced communication time up to 75 percent relative to MiniGhost’s default mapping on 128K cores of a Cray XK7 with sparse allocation. For the atmospheric modeling code E3SM/HOMME, our methods reduced communication time up to 31% on 16K cores of an IBM BlueGene/Q with contiguous allocation.

Abstract not provided.

Climate simulations with more accurate process-level representation at finer resolutions (<100 km) are a pressing need in order to provide more detailed actionable information to policy makers regarding extreme events in a changing climate. Computational limitation is a major obstacle for building and running high-resolution (HR, here 0.25° average grid spacing at the Equator) models (HRMs). A more affordable path to HRMs is to use a global regionally refined model (RRM), which only simulates a portion of the globe at HR while the remaining is at low resolution (LR, 1°). In this study, we compare the Energy Exascale Earth System Model (E3SM) atmosphere model version 1 (EAMv1) RRM with the HR mesh over the contiguous United States (CONUS) to its corresponding globally uniform LR and HR configurations as well as to observations and reanalysis data. The RRM has a significantly reduced computational cost (roughly proportional to the HR mesh size) relative to the globally uniform HRM. Over the CONUS, we evaluate the simulation of important dynamical and physical quantities as well as various precipitation measures. Differences between the RRM and HRM over the HR region are predominantly small, demonstrating that the RRM reproduces the precipitation metrics of the HRM over the CONUS. Further analysis based on RRM simulations with the LR vs. HR model parameters reveals that RRM performance is greatly influenced by the different parameter choices used in the LR and HR EAMv1. This is a result of the poor scale-aware behavior of physical parameterizations, especially for variables influencing sub-grid-scale physical processes. RRMs can serve as a useful framework to test physics schemes across a range of scales, leading to improved consistency in future E3SM versions. Applying nudging-to-observations techniques within the RRM framework also demonstrates significant advantages over a free-running configuration for use as a test bed and as such represents an efficient and more robust physics test bed capability. Our results provide additional confirmatory evidence that the RRM is an efficient and effective test bed for HRM development.

We present an architecture-portable and performant implementation of the atmospheric dynamical core (High-Order Methods Modeling Environment, HOMME) of the Energy Exascale Earth System Model (E3SM). The original Fortran implementation is highly performant and scalable on conventional architectures using the Message Passing Interface (MPI) and Open MultiProcessor (OpenMP) programming models. We rewrite the model in C++ and use the Kokkos library to express on-node parallelism in a largely architecture-independent implementation. Kokkos provides an abstraction of a compute node or device, layout-polymorphic multidimensional arrays, and parallel execution constructs. The new implementation achieves the same or better performance on conventional multicore computers and is portable to GPUs. We present performance data for the original and new implementations on multiple platforms, on up to 5400 compute nodes, and study several aspects of the single-and multi-node performance characteristics of the new implementation on conventional CPU (e.g., Intel Xeon), many core CPU (e.g., Intel Xeon Phi Knights Landing), and Nvidia V100 GPU.

Abstract not provided.

This work documents the first version of the U.S. Department of Energy (DOE) new Energy Exascale Earth System Model (E3SMv1). We focus on the standard resolution of the fully coupled physical model designed to address DOE mission-relevant water cycle questions. Its components include atmosphere and land (110-km grid spacing), ocean and sea ice (60 km in the midlatitudes and 30 km at the equator and poles), and river transport (55 km) models. This base configuration will also serve as a foundation for additional configurations exploring higher horizontal resolution as well as augmented capabilities in the form of biogeochemistry and cryosphere configurations. The performance of E3SMv1 is evaluated by means of a standard set of Coupled Model Intercomparison Project Phase 6 (CMIP6) Diagnosis, Evaluation, and Characterization of Klima simulations consisting of a long preindustrial control, historical simulations (ensembles of fully coupled and prescribed SSTs) as well as idealized CO2 forcing simulations. The model performs well overall with biases typical of other CMIP-class models, although the simulated Atlantic Meridional Overturning Circulation is weaker than many CMIP-class models. While the E3SMv1 historical ensemble captures the bulk of the observed warming between preindustrial (1850) and present day, the trajectory of the warming diverges from observations in the second half of the twentieth century with a period of delayed warming followed by an excessive warming trend. Using a two-layer energy balance model, we attribute this divergence to the model's strong aerosol-related effective radiative forcing (ERFari+aci = −1.65 W/m2) and high equilibrium climate sensitivity (ECS = 5.3 K).

A set of algorithms based on characteristic discontinuous Galerkin methods is presented for tracer transport on the sphere. The algorithms are designed to reduce message passing interface communication volume per unit of simulated time relative to current methods generally, and to the spectral element scheme employed by the U.S. Department of Energy's Exascale Earth System Model (E3SM) specifically. Two methods are developed to enforce discrete mass conservation when the transport schemes are coupled to a separate dynamics solver; constrained transport and Jacobian-combined transport. A communication-efficient method is introduced to enforce tracer consistency between the transport scheme and dynamics solver; this method also provides the transport scheme's shape preservation capability. A subset of the algorithms derived here is implemented in E3SM and shown to improve transport performance by a factor of 2.2 for the model's standard configuration with 40 tracers at the strong scaling limit of one element per core.

Atmospheric modeling with element-based high-order Galerkin methods presents a unique challenge to the conventional physics–dynamics coupling paradigm, due to the highly irregular distribution of nodes within an element and the distinct numerical characteristics of the Galerkin method. The conventional coupling procedure is to evaluate the physical parameterizations (physics) on the dynamical core grid. Evaluating the physics at the nodal points exacerbates numerical noise from the Galerkin method, enabling and amplifying local extrema at element boundaries. Grid imprinting may be substantially reduced through the introduction of an entirely separate, approximately isotropic finite-volume grid for evaluating the physics forcing. Integration of the spectral basis over the control volumes provides an area-average state to the physics, which is more representative of the state in the vicinity of the nodal points rather than the nodal point itself and is more consistent with the notion of a “large-scale state” required by conventional physics packages. This study documents the implementation of a quasi-equal-area physics grid into NCAR’s Community Atmosphere Model Spectral Element and is shown to be effective at mitigating grid imprinting in the solution. The physics grid is also appropriate for coupling to other components within the Community Earth System Model, since the coupler requires component fluxes to be defined on a finite-volume grid, and one can be certain that the fluxes on the physics grid are, indeed, volume averaged.

Atmospheric tracer transport is a computationally demanding component of the atmospheric dynamical core of weather and climate simulations. Simulations typically have tens to hundreds of tracers. A tracer field is required to preserve several properties, including mass, shape, and tracer consistency. To improve computational efficiency, it is common to apply different spatial and temporal discretizations to the tracer transport equations than to the dynamical equations. Using different discretizations increases the difficulty of preserving properties. This paper provides a unified framework to analyze the property preservation problem and classes of algorithms to solve it. We examine the primary problem and a safety problem; describe three classes of algorithms to solve these; introduce new algorithms in two of these classes; make connections among the algorithms; analyze each algorithm in terms of correctness, bound on its solution magnitude, and its communication efficiency; and study numerical results. A new algorithm, QLT, has the smallest communication volume, and in an important case it redistributes mass approximately locally. These algorithms are only very loosely coupled to the underlying discretizations of the dynamical and tracer transport equations and thus are broadly and efficiently applicable. In addition, they may be applied to remap problems in applications other than tracer transport.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

It is the purpose of this paper to provide a comprehensive documentation of the new NCAR (National Center for Atmospheric Research) version of the spectral element (SE) dynamical core as part of the Community Earth System Model (CESM2.0) release. This version differs from previous releases of the SE dynamical core in several ways. Most notably the hybrid sigma vertical coordinate is based on dry air mass, the condensates are dynamically active in the thermodynamic and momentum equations (also referred to as condensate loading), and the continuous equations of motion conserve a more comprehensive total energy that includes condensates. Not related to the vertical coordinate change, the hyperviscosity operators and the vertical remapping algorithms have been modified. The code base has been significantly reduced, sped up, and cleaned up as part of integrating SE as a dynamical core in the CAM (Community Atmosphere Model) repository rather than importing the SE dynamical core from High-Order Methods Modeling environment as an external code.