4.3. Full Tangent Preconditioner

In addition to the nodal preconditioners, Sierra/SM provides the capability to use a full tangent preconditioner in the CG algorithm. The full tangent preconditioner employs a scalable parallel linear solver to solve for the gradient direction using the full tangent stiffness matrix. Although the full tangent preconditioner is significantly more costly per CG iteration than the nodal preconditioners, it can solve problems in a smaller number of iterations. Full tangent preconditioners are especially effective in solving poorly conditioned problems that are often extremely difficult to solve using the CG algorithm with nodal preconditioners, such as those problems that involve the bending response of long, slender members.

To use a full tangent preconditioner generally two command blocks are added to the input file, one required and one optional. First, the FULL TANGENT PRECONDITIONER command block, described in this section, must be added to the CG command block. The FULL TANGENT PRECONDITIONER command block is used instead of the PRECONDITIONER command line in the CG command block. Second, a command block for a linear solver may be defined in the SIERRA scope, as described in Section 4.4. That linear solver command block may then be referenced from within the FULL TANGENT PRECONDITIONER command block, instructing the CG algorithm to use the specified linear solver. If no linear solver command block is given and/or no linear solver is referenced in the FULL TANGENT PRECONDITIONER command block, the FETI linear solver will be employed with default settings. The command block to enable the full tangent preconditioner is as follows:

Known Issue

Deactivation of element blocks (see Section 6.1.7.9) in conjunction with the full tangent preconditioner is available but not yet fully tested. Use caution when combining these capabilities.

BEGIN FULL TANGENT PRECONDITIONER
  #
  # solver selection commands
  LINEAR SOLVER = <string>linear_solver_name(FETI)
    [DURING <string list>period_names]
  NODAL PRECONDITIONER METHOD = ELASTIC|PROBE|DIAGONAL|BLOCK
    (ELASTIC)
  #
  # tangent matrix formation commands
  PROBE FACTOR = <real>probe_factor(1.0e-6)
    [DURING <string list>period_names]
  BALANCE PROBE = <integer>balance_probe(1)
    [DURING <string list>period_names]
  PENALTY FACTOR = <real>penalty_factor(100.0)
    [DURING <string list>period_names]
  # useful to improve ill-conditioned tangent matrix
  TANGENT DIAGONAL SCALE = <real>tangent_diag_scale(0.0)
    [DURING <string list>period_names]
  TANGENT DIAGONAL SHIFT = <real>tangent_diag_shift(0.0)
    [DURING <string list>period_names]
  # turn off for speed
  CONDITIONING = NO_CHECK|CHECK|AUTO_REGULARIZATION(CHECK)
    [DURING <string list>period_names]
  HANGING NODE FACTOR = <real>(1.0e-3)
    [DURING <string list>period_names]
  #
  # reset and iteration commands
  MAXIMUM RESETS FOR MODELPROBLEM = <integer>max_mp_resets
    (100000) [DURING <string list>period_names]
  MAXIMUM RESETS FOR LOADSTEP = <integer>max_ls_resets
    (100000) [DURING <string list>period_names]
  MAXIMUM ITERATIONS FOR MODELPROBLEM
    = <integer>max_mp_iter(100000)
    [DURING <string list>period_names]
  MAXIMUM ITERATIONS FOR LOADSTEP = <integer>max_ls_iter
    (100000) [DURING <string list>period_names]
  ITERATION UPDATE = <integer>iter_update
    [DURING <string list>period_names]
  ITERATION PRINT = <integer>iprint
  # freeze current preconditioner
  SMALL NUMBER OF ITERATIONS = <integer>small_num_iter
    [DURING <string list>period_names]
  SMOOTHING STEPS = <integer>smooth_steps(0)
    [DURING <string list>period_names]
  MINIMUM SMOOTHING ITERATIONS = <integer>min_smooth_iter(0)
    [DURING <string list>period_names]
  MAXIMUM SMOOTHING ITERATIONS = <integer>max_smooth_iter(0)
    [DURING <string list>period_names]
  TARGET SMOOTHING RESIDUAL = <integer>tgt_smooth_resid(0)
    [DURING <string list>period_names]
  TARGET SMOOTHING RELATIVE RESIDUAL
    = <integer>tgt_smooth_rel_resid(0)
    [DURING <string list>period_names]
  #
  # fall-back strategy commands
  STAGNATION THRESHOLD = <real>stagnation(1.0e-12)
    [DURING <string list>period_names]
  # triggers preconditioner switch or update
  MINIMUM CONVERGENCE RATE = <real>min_conv_rate(1.0e-4)
    [DURING <string list>period_names]
  # either switch to nodal preconditioner or update
  ADAPTIVE STRATEGY = SWITCH|UPDATE|NONE(SWITCH)
    [DURING <string list>period_names]


END [FULL TANGENT PRECONDITIONER]

The FULL TANGENT PRECONDITIONER command block contains all command lines related to the interaction with the linear solver. There are command lines to control selection of the linear solver, formation of the matrices passed into the linear solver, CG iteration strategies, updating strategies, and strategies for dealing with poor convergence. Reasonable defaults have been set for most problems. The command lines in this command block are described in Section 4.3.1 through Section 4.3.5.

4.3.1. Solver Selection Commands

LINEAR SOLVER = <string>linear_solver_name
  [DURING <string list>period_names]
NODAL PRECONDITIONER METHOD = ELASTIC|PROBE|DIAGONAL|
  BLOCK(ELASTIC)

The command lines listed above are used in selecting a linear solver for use with the full tangent preconditioner and in selecting a nodal preconditioner. The LINEAR SOLVER command line specifies the name of the linear solver used to compute the action of the preconditioner by solving the linear system. If no linear solver is specified, Sierra/SM will default to using the FETI linear solver with its default settings. Although several linear-solver packages are available for use as preconditioners in Sierra/SM, we recommend that the FETI equation solver be used in production analyses and is actively maintained and tested by the Sierra/SM development team. Other linear solvers may have compatibility problems with other code capabilities such as MPCs and contact.

In order to use a non-default linear solver, the user will need to define a solver in the SIERRA scope. For an example on how to set up a custom FETI EQUATION SOLVER command block, see Section 4.4

In some cases, large (over 100,000 elements) analyses may see a 4x or greater time speedup on GPU-accelerated platforms by specifying a GDSW solver. The GDSW equation solver is actively maintained and tested by the SIERRA/SD development team. An example code block defining and calling a GDSW solver with a CG preconditioner is shown below.

...
    begin solver
      begin cg
        target relative residual = 1.0e-09
        begin full tangent preconditioner
          linear solver = my_gdsw
        end
      end
    end
  end adagio region adagio
end adagio procedure Agio_Procedure

begin gdsw equation solver my_gdsw
end

An arbitrary number of equation solver command blocks can be included in the input file to define various sets of solver options. Each equation solver command block must be given a unique name, which is referenced in the linear_solver_name parameter of the LINEAR SOLVER command line. This name selects the specified linear solver as a preconditioner for the CG solver.

When Sierra/SM uses a full tangent preconditioner, it also forms a nodal preconditioner. This nodal preconditioner is used in contact calculations, as a fall-back preconditioner if the full tangent preconditioner does not perform well and optionally for performing smoothing iterations before use of the full tangent preconditioner.

The nodal preconditioner is selected with the NODAL PRECONDITIONER METHOD command line. Four options are available: ELASTIC, PROBE, DIAGONAL, and BLOCK. The ELASTIC option is the default. These options have the same meaning as the corresponding options given for the nodal preconditioner in the CG command block, as documented in Section 4.2.2.

4.3.2. Matrix Formation Commands

PROBE FACTOR = <real>probe_factor(1.0e-6)
  [DURING <string list>period_names]
BALANCE PROBE = <integer>balance_probe(1)
  [DURING <string list>period_names]
PENALTY FACTOR = <real>penalty_factor(100.0)
  [DURING <string list>period_names]
TANGENT DIAGONAL SCALE = <real>tangent_diag_scale(0.0)
  [DURING <string list>period_names]
TANGENT DIAGONAL SHIFT = <real>tangent_diag_shift(0.0)
  [DURING <string list>period_names]
CONDITIONING = NO_CHECK|CHECK|AUTO_REGULARIZATION(CHECK)
  [DURING <string list>period_names]
HANGING NODE FACTOR = <real>(1.0e-3)
  [DURING <string list>period_names]

The command lines listed above can be used to optionally control the way the stiffness matrix is formed. This matrix is formed by assembling contributions from all active elements. These element stiffness matrices are formed by finite differencing.

The PROBE FACTOR command line controls the finite difference probing distance relative to element size. The default value of probe_factor is 1.0e-6. Smaller values may give a better tangent approximation, but they may also generate round-off errors.

The BALANCE PROBE command line selects the type of finite-differencing scheme used to probe for the stiffness. The value of balance_probe can be set to 0, 1, or 2. Setting balance_probe to 0 causes the preconditioner to be formed by forward finite differencing. Probing occurs in only one direction, resulting in first-order truncation error in the preconditioner. Setting balance_probe to 1, the default for the full tangent preconditioner, causes the preconditioner to be formed with central differencing. Probing at each element degree of freedom is performed in both positive and negative directions, leading to a second-order estimate of the tangent stiffness. The value of balance_probe may also be set to 2, which allows the full tangent preconditioner to obtain central finite differencing with fourth-order error. This scheme requires twice the work as the central-differencing scheme but is more accurate. See Section 4.2.2 for a more detailed discussion of these options.

Constraints (i.e., contact, rigid body, and periodic boundary conditions) are enforced by conversion to penalty “elements” whose stiffnesses (controlled by the PENALTY FACTOR command line—see below) are contributed to the linear system directly. The errors caused by penalty compliance are corrected in each CG iteration, so the converged solution will not have errors due to penalty compliance.

The PENALTY FACTOR command line specifies the penalty stiffness. At each constraint, the diagonal tangent stiffness of the side B degree of freedom is multiplied by the value of the penalty_factor parameter. The default value for this parameter is 100.0, which is reasonable for most problems. Assigning a high value (greater than about 1.0e+8) can cause poor conditioning of the linear system and also cause the linear solver to perform poorly. Assigning a low value (less than about 10) could require more CG iterations due to the correction of errors resulting from penalty compliance.

The TANGENT DIAGONAL SCALE command line specifies a tangent matrix diagonal scale factor, which is used to scale the diagonal entries of the tangent matrix. This is useful in some situations where a model does not have sufficient boundary conditions to remove all rigid body modes. This command has also been used as a last resort strategy to achieve convergence when all else fails. The diagonal terms of the tangent matrix are scaled by \((1.0 + \texttt{\small tangent\_diagonal\_scale})\). The default value of tangent_diagonal_scale is 0.0, which means the tangent matrix diagonals are not scaled. A value of 0.1 scales up all tangent matrix diagonal terms by 10 percent.

The TANGENT DIAGONAL SHIFT command line specifies a tangent matrix diagonal shift, which is summed into the diagonal entries of the tangent matrix. This is useful in some situations where a model does not have sufficient boundary conditions to remove all rigid body modes. This command has also been used as a last resort strategy to achieve convergence when all else fails. The diagonal terms of the tangent matrix are shifted by \(( \texttt{\small num\_nodes} * \texttt{\small tangent\_diagonal\_shift})\). The default value of tangent_diagonal_shift is 0.0, which means the tangent matrix diagonals are not shifted.

The CONDITIONING command line controls the linear system conditioning inside the FETI solver. It has no effect on any other linear system solver. By default, when used with the full tangent preconditioner FETI checks the conditioning of its coarse level matrix and reports a warning to the log file if it finds ill-conditioning (defined as a condition number greater than \(1\times 10^5\), where the condition number is defined as the ratio of the maximum eigenvalue to the minimum eigenvalue). This default behavior corresponds to the CHECK option. For performance optimization this check may be turned off using the NO_CHECK option. The third option, AUTO_REGULARIZATION, instructs FETI to add a small shift to its coarse matrix to regularize the linear system and artificially reduce the condition number. In the context of the CG solver, this will affect convergence but not the residual calculations. Coarse level matrix ill-conditioning often indicates an issue with the problem, such as the presence of zero energy rigid body modes either inherently or temporarily as a result of a CG iteration, and AUTO_REGULARIZATION may improve CG convergence in the presence of ill-conditioning. A basic power method is used to compute the highest eigenvalue, and a shifted power method is used to compute the lowest eigenvalue. This capability should be viewed both as a diagnostic tool and a robustness aide.

The HANGING NODE FACTOR command is used to apply a nodal stiffness to any nodes that may exist on a processor but are not attached to any element on that processor. This configuration can occur with parallel contact, spot welds, MPCs, and other non-local joints and boundary conditions. This artificial stiffness is used to avoid singularities in the processor local stiffness matrix. The value used is a scale factor on the diagonal nodal stiffness.

4.3.3. Reset and Iteration Commands

MAXIMUM RESETS FOR MODELPROBLEM = <integer>max_mp_resets
  (100000) [DURING <string list>period_names]
MAXIMUM RESETS FOR LOADSTEP = <integer>max_ls_resets
  (100000) [DURING <string list>period_names]
MAXIMUM ITERATIONS FOR MODELPROBLEM = <integer>max_mp_iter
  (100000) [DURING <string list>period_names]
MAXIMUM ITERATIONS FOR LOADSTEP = <integer>max_ls_iter
  (100000) [DURING <string list>period_names]
ITERATION UPDATE = <integer>iter_update
  [DURING <string list>period_names]
ITERATION PRINT = <integer>iprint
SMALL NUMBER OF ITERATIONS = <integer>small_num_iter(0)
  [DURING <string list>period_names]

When a linear solver is used as a preconditioner, the efficiency of the solution can often be dramatically affected by the frequency at which the preconditioner is updated. The computational expense of updating the preconditioner, which involves forming and factorizing the stiffness matrix, is often much higher than the cost of applying the preconditioner, which involves only a back-solve, during a CG iteration. If the stiffness does not dramatically change from iteration to iteration, it is often much more efficient to re-use the preconditioner for many iterations, or even load steps, than it is to update it at every iteration.

The command lines listed above can be used to control when the preconditioner is updated. As Sierra/SM allows for the nodal preconditioner to be used in place of the full tangent preconditioner in some situations, these command lines also control this behavior. These command lines can be appended with the DURING specification, as discussed in Section 4.2.1.

The MAXIMUM RESETS FOR MODELPROBLEM command line limits the number of times that the preconditioner can be reset (updated) during a model problem. By default, there is no limit on the number of updates that can occur during a model problem. The MAXIMUM RESETS FOR LOADSTEP command line limits the updates that can occur during a load step. By default, there is also no limit on the number of updates that can occur during a load step.

The MAXIMUM ITERATIONS FOR LOADSTEP and MAXIMUM ITERATIONS FOR MODELPROBLEM command lines can be used to limit the number of CG iterations performed using the full tangent preconditioner per model problem or load step, respectively. The values of the parameters max_mp_iter and max_ls_iter are both unlimited by default. If one of these iteration limits is reached, the CG solver will continue to iterate using the fall-back nodal preconditioner until either the CG solver iteration limit is reached or convergence is achieved. It can be useful to limit the iterations taken by the full tangent preconditioner. If the full tangent preconditioner is performing poorly, the nodal preconditioner may work better sometimes.

The ITERATION UPDATE command line controls the frequency at which the full tangent preconditioner is updated. By default, it is only updated at the first iteration of each model problem. If this command line is used, the preconditioner will be updated at the frequency specified by iter_update. If a model experiences highly nonlinear behavior over the course of a model problem, it may be useful to use this command line to cause more frequent preconditioner updates. Such problems may converge better when iter_update is set to about 10. Realize that frequent updates may reduce the number of CG iterations but dramatically increase the computational cost.

The ITERATION PRINT command line controls the frequency at which the convergence information is printed to the log file while applying the full tangent preconditioner. The default behavior is inherited from the same command in the containing CG block, which defaults to printing every iteration when the full tangent preconditioner is in use. Alternatively, when a large number of conjugate gradient smoothing iterations are being used in conjunction with the full tangent preconditioner it can be convenient to set ITERATION PRINT to a larger number in the CG block while also setting ITERATION PRINT to a smaller number in the full tangent preconditioner. Iteration information will also be output during a full tangent preconditioner update.

It can often be efficient to update the preconditioner infrequently. The SMALL NUMBER OF ITERATIONS command line is used to “freeze” the preconditioner, or prevent updates, until a model problem requires more iterations than the number specified in small_num_iter. The default value of small_num_iter is 0, meaning that the preconditioner is never re-used for the next model problem unless the SMALL NUMBER OF ITERATIONS command line is used. It is often beneficial to set small_num_iter to 10. This can result in the re-use of the preconditioner over many load steps. When this command line is used, the CG solver may converge in a small number of iterations immediately following a preconditioner update; however, over the course of several load steps, the iteration counts slowly increase because the preconditioner is out of date. At a certain point, the iteration count exceeds small_num_iter, and the preconditioner is reset. This solution strategy is efficient because it allows re-use of the preconditioner while it is still relatively effective.

Warning

The SMALL NUMBER OF ITERATIONS command is not guaranteed to be compatible with linear solvers other than FETI. For instance, the Aztec solver fails to find the underlying preconditioner when it is used in an aria and an adagio region. A possible workaround is to use a unique linear solver in each region.

4.3.4. Smoothing Iterations

SMOOTHING STEPS = <integer>smooth_steps(0)
  [DURING <string list>period_names]
MINIMUM SMOOTHING ITERATIONS = <integer>min_smooth_iter(0)
  [DURING <string list>period_names]
MAXIMUM SMOOTHING ITERATIONS = <integer>max_smooth_iter(0)
  [DURING <string list>period_names]
TARGET SMOOTHING RESIDUAL = <integer>tgt_smooth_resid(0)
  [DURING <string list>period_names]
TARGET SMOOTHING RELATIVE RESIDUAL =
  <integer>tgt_smooth_rel_resid(0)
  [DURING <string list>period_names]
AUTOMATIC SMOOTHING FACTOR = <real>auto_smooth
AUTOMATIC SMOOTHING RELATIVE RESIDUAL FACTOR =
  <real>rel_auto_smooth
AUTOMATIC SMOOTHING RESIDUAL FACTOR = <real>abs_auto_smooth

Sometimes it is beneficial to perform many CG iterations using the nodal preconditioner before applying the full tangent preconditioner. The iterations with the nodal preconditioner are used as a “smoother” to put the model in a better state for the full tangent preconditioner. The option SMOOTHING STEPS will delay the use of the full tangent preconditioner until the specified number of model problems have been solved, in a multilevel solver a contact iteration is a model problem solve. Smoothing iterations are controlled using the MINIMUM SMOOTHING ITERATIONS, MAXIMUM SMOOTHING ITERATIONS, TARGET SMOOTHING RESIDUAL, and TARGET SMOOTHING RELATIVE RESIDUAL command lines. By default, no smoothing iterations are taken. Using these commands, the user can specify that smoothing iterations should be taken and control how many are taken.

The number of smoothing iterations taken can vary based on the convergence of the solver during those iterations. The smoothing iteration commands allow the user to control the desired convergence of the smoother before switching to the full tangent preconditioner. These commands behave the same way as the standard solver convergence commands, except that they apply only to the smoothing iterations. The standard convergence criteria still must be met to obtain a converged solution. The MINIMUM SMOOTHING ITERATIONS command controls the minimum smoothing iterations taken, and the MAXIMUM SMOOTHING ITERATIONS command controls the maximum smoothing iterations. The TARGET SMOOTHING RESIDUAL and TARGET SMOOTHING RELATIVE RESIDUAL control the residual and relative residual, respectively, that must be obtained in the smoothing iterations before switching to the full tangent preconditioner. Once tgt_smooth_resid has been reached, the solver switches from using the nodal preconditioner to the tangent preconditioner. If max_smooth_iter is reached before the target smoothing convergence is reached, the solver switches to using the full tangent preconditioner.

This feature can also be used in cases where the nodal preconditioner may be effective enough to achieve convergence with relatively few iterations, but occasionally the nodal preconditioner is not effective. If max_smooth_iter is set to a relatively high number, many load steps can be solved without ever using the full tangent preconditioner, but this preconditioner is used on the more difficult steps that require more iterations.

Another use of the smoothing iteration control commands is for switching between the nodal preconditioner and the full tangent preconditioner for certain time periods. These command lines, like all the other command lines discussed above, can be set on a period-specific basis by appending them with the DURING keyword, followed by a list of periods. To use the nodal preconditioner on some periods, the user can set max_smooth_iter to a high number for those periods and leave the default value of 0 for the periods in which the full tangent preconditioner is desired.

An alternative way the smoothing tolerances can be set is with the AUTOMATIC SMOOTHING FACTOR commands. The automatic smoothing factor commands automatically select an appropriate smoothing residual target based on the initial and target residuals.

The AUTOMATIC SMOOTHING RELATIVE RESIDUAL FACTOR effectively sets the TARGET SMOOTHING RELATIVE RESIDUAL \(r_S\) based off of the initial relative residual \(r_0\), target relative residual \(r_T\) and input rel_auto_smooth scaling factor \(f\) using the following equation.

\[r_S = e^{f*(ln(r_T)-ln(r_0))+ln(r_0)}\]

Similarly The AUTOMATIC SMOOTHING RESIDUAL FACTOR effectively sets the TARGET SMOOTHING RESIDUAL \(R_S\) based off of the initial residual \(R_0\), target residual \(R_T\) and input abs_auto_smooth scaling factor \(F\) using the following equation.

\[R_S = e^{F*(ln(R_T)-ln(R_0))+ln(R_0)}\]

Use of the AUTOMATIC SMOOTHING FACTOR command sets both the AUTOMATIC SMOOTHING RELATIVE RESIDUAL FACTOR and the AUTOMATIC SMOOTHING RESIDUAL FACTOR at the same time to the same value.

When using the AUTOMATIC SMOOTHING FACTOR command the scaling factor may be any number between zero and one though appropriate values are generally in the range of 0.1 to 0.5.

4.3.5. Fall-Back Strategy Commands

STAGNATION THRESHOLD = <real>stagnation(1.0e-12)
MINIMUM CONVERGENCE RATE = <real>min_conv_rate(1.0e-4)
ADAPTIVE STRATEGY = SWITCH|UPDATE|NONE(SWITCH)

Occasionally, the full tangent preconditioner may stagnate, failing to significantly reduce the residual from one CG iteration to the next. At each iteration, Sierra/SM checks for slow convergence, and attempts to remedy poor convergence if detected. The commands listed above can optionally be added to the FULL TANGENT PRECONDITIONER command block.

The STAGNATION THRESHOLD and MINIMUM CONVERGENCE RATE command lines are used to set the stagnation and min_conv_rate parameters, respectively. These parameters are used in a strategy that attempts to remedy slow convergence. The method for remedying slow convergence is controlled with the ADAPTIVE STRATEGY command line, which can have a value of SWITCH, UPDATE, or NONE. The default strategy is to SWITCH from the tangent preconditioner to the nodal preconditioner. The formula for computing the convergence rate is given by

\[\mathrm{abs}( ( ||R_{n-1}|| - ||R_n|| ) / ||R_{n-1}|| )\]

where \(R\) is the residual force and \(n\) is the iteration counter. In the SWITCH strategy, if the convergence rate is less than the value of min_conv_rate but greater than the value of stagnation, the CG solver will switch to using the fall-back nodal preconditioner. In the UPDATE strategy, the full tangent preconditioner would be updated instead. With either option SWITCH or UPDATE, if the convergence rate is below the value of stagnation, the solver will switch to the nodal preconditioner. The adaptive strategy can be turned off via the NONE option. When using this option, the iterations will continue with the existing preconditioner until an update or switch is performed by some other means.

4.3.6. Log File Output

The full tangent preconditioner produces a few useful markings in the log file while solving a model problem. The following markings may appear in the “LINEAR ITER” column to indicate various conditions during the solve:

Table 4.2 Full Tangent Preconditioner Log File Markings

Mark	Description
U	The preconditioner was updated this iteration. This iteration required a linear system solve, and may take much longer than subsequent iterations.
S	The preconditioner stagnated on this iteration and could not successfully reach a solution. Note, this overrides the “U’’” marker.
M	The preconditioner reached the maximum linear solve iterations (default 500) without reaching the target residual. This often appears as “M500” and can indicate incorrect problem setup such as the presence of rigid body modes in a quasistatic problem.

4.3.7. CG with Full Tangent Preconditioner Input Example

begin solver
  begin cg
    target relative residual = 1.0e-06
    Maximum Iterations       = 100
    # uses FETI as the default linear equation solver
    begin full tangent preconditioner
      tangent diagonal scale = 1e-6
    end
  end
end