POR-SALSA: A Numerical Simulator for Nonisothermal Multiphase Subsurface Flow on Massively Parallel Architectures

Motivation and Applications

Realistic numerical simulation of multiphase flow in complex geological is hampered in many cases by limited model resolution, difficulties in upscaling of heterogeneous property distributions, and the large range of time scales presented by the physical processes involved. Recent advances in massively parallel computing architectures and algorithms at Sandia National Laboratories has enabled the development of an advanced research and development code for efficient simulation of very large nonisothermal, multiphase flow problems. This code provides a tool for investigation and analysis of a wide range of geoscience problems including:

• Radioactive and Hazardous Waste Disposal
• Design and Interpretation of Supporting Experiments
• Basin-Scale Analysis of Groundwater Resources
• Subsurface Remediation and Site Assessment
• Enhanced Oil and Gas Recovery and Reservoir Engineering
• Geothermal Reservoir Engineering

It is anticipated that POR-SALSA will be capable of running fully transient simulations with 108 node resolution on next generation massively parallel machines such as the 9,000 node Sandia teraflops recently built by Intel for Sandia National Laboratories. While this code is targeted primarily toward massively parallel machines, it is written using MPI message passing protocols which enable it to run on networked work stations.

Spatial discretization of the "Dryout Scenario Simulation" at the Yucca Mountain Project. All major hydrostratigraphic units are represented. The spatial domain is approximately 3.0 km x 1.7 km x 0.7 km and the mesh was constructed from general, irregular hexadecimal elements using CUBIT (Blacker et. al., 1994). Over 358,000 node points were used in this descretization.

Current Capabilities and Features

Current capabilities of the Sandia massively parallel flow simulator include nonisothermal two-phase (liquid-gas) multicomponent (air-water) flow, including capillary pressure effects; binary diffusion in the gas phase; and conductive, latent, and sensible heat transport (Martinez et al., 1997). Numerical implementation utilizes a finite element approach, allowing fully general, unstructured grids that allow simulation of complex geologic features such as faults, offsets, and pinchouts. This also enables simulation of high resolution features associated with engineered subsurface barriers. Control volume finite elements have also been implemented to aid in the transition between one and two phase states. Message passing and domain decomposition are used for implementing a scalable algorithm for distributed memory computers built on the MPSalsa code platform developed at Sandia (Shadid et al., 1996). The parallel implementation is enhanced by a persistent variable solution approach that minimizes the need for subsidiary calculations when single and two-phase states are encountered at different nodes in the system.

Other numerical features include dynamic time-stepping algorithms (predictor/corrector), Newton and modified Newton iteration schemes to handle non-linearities, and coupling with the AZTEC parallel solver library developed at Sandia. Geostatistically-based fields of heterogeneous material properties have been incorporated by linking the geostatistical software library, GSLIB, with the mesh generating software and input package used for the code.

Near term upgrades include leveraging the contaminant transport simulation capabilities of MPSalsa. This will be supplemented by incorporating a multirate mass transfer formulation (Haggerty and Gorelick, 1995) as well as radionuclide chain transport modeling.

Performance

A range of test problems have been run, indicating a speed up efficiency of 79% on 1,024 processors for an unstructured grid composed of 358,000 grid points and 1.1 million unknowns. In real terms, this simulation required only 2 hours of CPU time where it is anticipated that a scalar machine would have required nearly 66 days.

Parallel processing performance data for a 3-dimensional nonisothermal test problem run on 256, 500, and 1024 processors. Each simulation required 71 time steps and 125 Newton iterations demonstrating fully parallel solver implementation.

Simulation of Gas Migration Test

Simulated Pressure Field at 24 Days

Observations

• Preferential release of gas pressure along shear zone.
• Entry pressures for concrete and sand / bentonite layers have been exceeded.
• Pressure shadow is evident above engineered barrier system due to low permeability plug.
• High resolution 3D geometry is required to capture the complex behavior.

Simulation of Dryout Scenario at the Yucca Mountain Project

Simulated Pressure Field at 23 Years 358,000 Elements 1.1 Million Unknowns 3.0 x 1.6 x 0.7 Kilometer Domain

2 Hours CPU Time vs. ~65.8 Days for a Single Processor 71 Time Steps 125 Newton Iterations 10,103 Linear Iterations Simulated Temperature Field at 23 Years

Observations

• Gas pressure build-up is moderate, and variation in gas pressure modest due to high relative permeability to the gas phase.
• Temperature distribution is very nearly at thermodynamically-saturated state.
• Air partial pressure down to 8% mole fraction

Saturation distribution in 1.2 kilometer east-west cross section of Yucca Mountain and infiltration flow lines. Blow up (left) shows near complete lateral diversion due to capillary barrier effect of a very thin geologic subunit.

References

Haggerty, R., and S. M. Gorelick, 1995, Multiple-Rate Mass Transfer for Modeling Diffusion and Surface Reactions in Media with Pore-scale Heterogeneity, Water Resources Res., 31 (10), 2383-2400.

Martinez, M. J., P. L. Hopkins, and J. N. Shadid, 1997, LDRD Final Report: Physical Simulation of Nonisothermal Multiphase Multicomponent Flow in Porous Media, Sandia National Laboratories, SAND97-1766.

Shadid, J. N., H. K. Moffat, S. A. Hutchinsons, G. L. Hennigan, K. D. Devine, and A. G. Salinger, 1996. MPSALSA, A Finite Element Computer Program for Reacting Flow Problems, Part I - Theoretical Development. Sandia National Laboratories, SAND95-2752.

Contacts