5.2.5.1. Teko Solver for OMD with Porous Flows Example
This example shows how to set up a Teko solver for an OMD example problem with porous flows.
5.2.5.1.1. Problem Files
The files required for this example can be downloaded here, or found at $TESTS/aria_rtest/example/training/omd_fic_enclosure (where TESTS=/projects/sierra/tests in a CEE environment, or your Sierra test distribution otherwise).
This example problem simulates a two dimensional foam-in-a-can experimental setup with enclosure radiation. There are three materials in this example: a fluid-only region (block 6), foam - a porous medium containing both fluid and solid phases (block 5), and a solid material for the can (blocks 1-4). For this particular case, a single Aria region encompassing the entire domain is used. A single, monolithic equation system is used for this simulation. Our main concern in this example, therefore, is demonstrating how to construct a Teko block preconditioner using the advice from the Road-map for Constructing Teko Solvers for the monolithic, multiphysics system.
5.2.5.1.2. Setting up a Teko Solver
Here we follow steps 1-4 as outlined in the Road-map for Constructing Teko Solvers.
We start by gather the DOFs in the monolithic linear system, which in alphabetical order are:
ENTHALPY GAS_PHASE_MASS_FRACTION_CO2 GAS_PHASE_MASS_FRACTION_Organic GAS_PHASE_PRESSURE GAS_PHASE_TEMPERATURE MASS_FRACTION_CO2 MASS_FRACTION_Organic PRESSURE SOLID_PHASE_DENSITY SOLID_PHASE_MASS_FRACTION_FoamA SOLID_PHASE_MASS_FRACTION_FoamB SOLID_PHASE_TEMPERATURE TEMPERATURE VELOCITY
With the DOFs identified above, we start by assigning each DOF to its own block. We employ a Block Gauss-Seidel approach with a direct solver for each diagonal sub-block, as recommended in step 2. The sample input deck block is shown below:
begin tpetra equation solver gmres_teko begin gmres solver convergence tolerance = 1e-8 begin teko preconditioner define matrix subblock number=1 dof=enthalpy define matrix subblock number=2 dof=gas_phase_mass_fraction_co2 define matrix subblock number=3 dof=gas_phase_mass_fraction_organic define matrix subblock number=4 dof=gas_phase_pressure define matrix subblock number=5 dof=gas_phase_temperature define matrix subblock number=6 dof=mass_fraction_co2 define matrix subblock number=7 dof=mass_fraction_organic define matrix subblock number=8 dof=pressure define matrix subblock number=9 dof=solid_phase_density define matrix subblock number=10 dof=solid_phase_mass_fraction_foama define matrix subblock number=11 dof=solid_phase_mass_fraction_foamb define matrix subblock number=12 dof=solid_phase_temperature define matrix subblock number=13 dof=temperature define matrix subblock number=14 dof=velocity begin teko subblock solver direct_solver begin klu2 solver end end begin block gauss-seidel solver use default inverse direct_solver for subblocks end end end end tpetra equation solver
Now we need to identify the DOF-to-block mapping. We start with the one-to-one mapping from step 2 and run a single time-step and observe the solve:
Equation System all->main: * Step : Transient, Strategy: LINE_SEARCH, Time: 1.00e-04, Step: 1.00e-04 * Matrix: Solver: "gmres_teko", Unknowns: 12270, Nonzeros: 688594 * Mesh : Processor 0 of 1: 2396 of 2396 elems, 2514 of 2514 nodes * Computing View Factors for enclosure encl_Enc1 N O N L I N E A R L I N E A R T I M E ------------------------------- -------------------- ----------------------- Step SubItns Resid Delta Itns Status Resid Asm/Slv/SubItn Time ---- ------- -------- -------- ---- ------ -------- ----------------------- 1 1 2.16e+02 1.49e+01 93 ok 9.94e-09 7.2e-02/1.2e+00/2.8e-02 2 1 7.13e-03 4.67e-01 274 ok 9.44e-09 6.4e-02/1.8e+00/2.8e-02 3 1 4.46e-06 1.68e-05 149 ok 9.05e-09 6.4e-02/1.4e+00/2.2e-02 4 0 2.20e-06 1.21e-09 128 ok 9.79e-09 6.2e-02/1.3e+00/2.9e-04 Termination reason: 1.20724e-09 < nonlinear_correction_tolerance(1e-06)Considering that we are employing a direct solver for each of the sub-blocks, these iterations counts are too high. Keeping usage of our direct solver, we can start grouping different DOFs to the same sub-block. Through some trial-and-error and physical intuition, we realize that:
The enthalpy and pressure equations are strongly coupled
The solid phase and gas phase temperatures, along with the gas phase pressure, are strongly coupled
All other blocks, in comparison, are only ‘weakly’ coupled
This is incorporated into the following input deck:
begin tpetra equation solver gmres_teko begin gmres solver convergence tolerance = 1e-8 begin teko preconditioner define matrix subblock number=1 dof=pressure define matrix subblock number=1 dof=enthalpy define matrix subblock number=2 dof=solid_phase_temperature define matrix subblock number=2 dof=gas_phase_temperature define matrix subblock number=2 dof=gas_phase_pressure define matrix subblock number=3 dof=velocity define matrix subblock number=4 dof=temperature define matrix subblock number=5 dof=gas_phase_mass_fraction_co2 define matrix subblock number=6 dof=gas_phase_mass_fraction_organic define matrix subblock number=7 dof=mass_fraction_co2 define matrix subblock number=8 dof=mass_fraction_organic define matrix subblock number=9 dof=solid_phase_density define matrix subblock number=10 dof=solid_phase_mass_fraction_foama define matrix subblock number=11 dof=solid_phase_mass_fraction_foamb begin teko subblock solver direct_solver begin klu2 solver end end begin block gauss-seidel solver use default inverse direct_solver for subblocks end end end end tpetra equation solverUsing the input deck from above, we see that the iteration counts are significantly improved:
Equation System all->main: * Step : Transient, Strategy: LINE_SEARCH, Time: 1.00e-04, Step: 1.00e-04 * Matrix: Solver: "gmres_teko", Unknowns: 12270, Nonzeros: 688594 * Mesh : Processor 0 of 1: 2396 of 2396 elems, 2514 of 2514 nodes * Computing View Factors for enclosure encl_Enc1 N O N L I N E A R L I N E A R T I M E ------------------------------- -------------------- ----------------------- Step SubItns Resid Delta Itns Status Resid Asm/Slv/SubItn Time ---- ------- -------- -------- ---- ------ -------- ----------------------- 1 1 2.16e+02 1.48e+01 3 ok 8.72e-09 6.7e-02/7.0e-01/2.8e-02 2 1 7.13e-03 3.35e-04 3 ok 9.55e-09 6.3e-02/7.0e-01/2.8e-02 3 0 2.69e-06 9.04e-09 3 ok 4.19e-09 6.3e-02/6.9e-01/2.5e-04 Termination reason: 9.03819e-09 < nonlinear_correction_tolerance(1e-06)
Now that we can reach convergence in a few iterations with Block Gauss-Seidel, we must now try to remove the direct solver usage for each sub-block. Converting each sub-block one-at-a-time, we observe that the solid phase density and all mass fraction terms can be readily solved via Jacobi. That is, the following input deck yields a solver with the same iteration counts as the direct solver for those sub-blocks
begin tpetra equation solver gmres_teko begin gmres solver convergence tolerance = 1e-8 begin teko preconditioner define matrix subblock number=1 dof=pressure define matrix subblock number=1 dof=enthalpy define matrix subblock number=2 dof=solid_phase_temperature define matrix subblock number=2 dof=gas_phase_temperature define matrix subblock number=2 dof=gas_phase_pressure define matrix subblock number=3 dof=velocity define matrix subblock number=4 dof=temperature define matrix subblock number=5 dof=gas_phase_mass_fraction_co2 define matrix subblock number=6 dof=gas_phase_mass_fraction_organic define matrix subblock number=7 dof=mass_fraction_co2 define matrix subblock number=8 dof=mass_fraction_organic define matrix subblock number=9 dof=solid_phase_density define matrix subblock number=10 dof=solid_phase_mass_fraction_foama define matrix subblock number=11 dof=solid_phase_mass_fraction_foamb begin teko subblock solver direct_solver begin klu2 solver end end begin teko subblock preconditioner jacobi_pc begin jacobi preconditioner end end begin block gauss-seidel solver use default inverse direct_solver for subblocks use inverse jacobi_pc for subblock 5 use inverse jacobi_pc for subblock 6 use inverse jacobi_pc for subblock 7 use inverse jacobi_pc for subblock 8 use inverse jacobi_pc for subblock 9 use inverse jacobi_pc for subblock 10 use inverse jacobi_pc for subblock 11 end end end end tpetra equation solverWith the following results from the log file:
Equation System all->main: * Step : Transient, Strategy: LINE_SEARCH, Time: 1.00e-04, Step: 1.00e-04 * Matrix: Solver: "gmres_teko", Unknowns: 12270, Nonzeros: 688594 * Mesh : Processor 0 of 1: 2396 of 2396 elems, 2514 of 2514 nodes * Computing View Factors for enclosure encl_Enc1 N O N L I N E A R L I N E A R T I M E ------------------------------- -------------------- ----------------------- Step SubItns Resid Delta Itns Status Resid Asm/Slv/SubItn Time ---- ------- -------- -------- ---- ------ -------- ----------------------- 1 1 2.16e+02 1.48e+01 3 ok 8.72e-09 6.6e-02/6.9e-01/2.7e-02 2 1 7.13e-03 3.35e-04 3 ok 9.55e-09 6.4e-02/6.9e-01/2.7e-02 3 0 2.69e-06 9.04e-09 3 ok 4.19e-09 6.3e-02/6.7e-01/2.4e-04 Termination reason: 9.03819e-09 < nonlinear_correction_tolerance(1e-06)When this occurs, you can generally combine all of the blocks using Jacobi into a single, large block solves via Jacobi. This typically leads to the same iteration count and a faster solver time-to-solution. In addition, we assume that DD-ILU can be easily applied to the velocity and temperature equations. The same cannot be said, however, for the combined pressure-enthalpy block and the combined solid/gas phase temperature-gas phase pressure block. Using a diagnostic preconditioner as recommended in the Road-map for Constructing Teko Solvers, we observe that those two sub-blocks require either a better preconditioner or a sub-iteration scheme. Here, we choose the latter option. Combining all of these elements together, we construct our Teko preconditioner. The solver from the input file is shown below:
begin tpetra equation solver gmres_teko begin gmres solver convergence tolerance = 1e-8 begin teko preconditioner define matrix subblock number=1 dof=pressure define matrix subblock number=1 dof=enthalpy define matrix subblock number=2 dof=solid_phase_temperature define matrix subblock number=2 dof=gas_phase_temperature define matrix subblock number=2 dof=gas_phase_pressure define matrix subblock number=3 dof=velocity define matrix subblock number=4 dof=temperature define matrix subblock number=5 dof=gas_phase_mass_fraction_co2 define matrix subblock number=5 dof=gas_phase_mass_fraction_organic define matrix subblock number=5 dof=mass_fraction_co2 define matrix subblock number=5 dof=mass_fraction_organic define matrix subblock number=5 dof=solid_phase_density define matrix subblock number=5 dof=solid_phase_mass_fraction_foama define matrix subblock number=5 dof=solid_phase_mass_fraction_foamb begin teko subblock solver ddilu_solver begin gmres solver convergence tolerance = 1e-2 maximum iterations = 30 restart iterations = 30 begin dd-ilu preconditioner end end end begin teko subblock preconditioner ddilu_pc begin dd-ilu preconditioner end end begin teko subblock preconditioner jacobi_pc begin jacobi preconditioner end end begin block gauss-seidel solver use inverse ddilu_solver for subblock 1 use inverse ddilu_solver for subblock 2 use inverse ddilu_pc for subblock 3 use inverse ddilu_pc for subblock 4 use inverse jacobi_pc for subblock 5 end end end end tpetra equation solverRunning the first step and inspecting the logfile, we observe that we have a fast, robust block preconditioner:
Equation System all->main: * Step : Transient, Strategy: LINE_SEARCH, Time: 1.00e-04, Step: 1.00e-04 * Matrix: Solver: "gmres_teko", Unknowns: 12270, Nonzeros: 688594 * Mesh : Processor 0 of 1: 2396 of 2396 elems, 2514 of 2514 nodes * Computing View Factors for enclosure encl_Enc1 N O N L I N E A R L I N E A R T I M E ------------------------------- -------------------- ----------------------- Step SubItns Resid Delta Itns Status Resid Asm/Slv/SubItn Time ---- ------- -------- -------- ---- ------ -------- ----------------------- 1 1 2.16e+02 1.48e+01 4 ok 2.65e-09 7.3e-02/3.8e-01/2.3e-02 2 1 7.13e-03 3.35e-04 4 ok 2.69e-09 6.2e-02/3.6e-01/2.2e-02 3 0 2.64e-06 8.99e-09 4 ok 4.15e-09 7.0e-02/3.7e-01/2.4e-04 Termination reason: 8.98683e-09 < nonlinear_correction_tolerance(1e-06)
Now that we have completed steps 1-4 as outlined in the Road-map for Constructing Teko Solvers, we have constructed a good Teko preconditioner for our OMD multiphysics problem.