# 6.8. Mesh Rebalancing

Mesh rebalancing is a feature in Presto that may improve the efficiency of an analysis. Two command blocks can be used to control mesh rebalancing: `REBALANCE` and `ZOLTAN PARAMETERS`. The `REBALANCE` command block is required; the `ZOLTAN PARAMETERS` command block is optional. Sections Section 6.8.1 and Section 6.8.2 describe these command blocks.

## 6.8.1. Rebalance

    BEGIN REBALANCE
      ELEMENT GROUPING TYPE = SPLIT SPH AND STANDARD ELEMENTS|
        UNIFORM UNIFIED| SPLIT SPH AND DEM AND STANDARD ELEMENTS|
        SPLIT DEM AND STANDARD ELEMENTS |
        SPLIT PARTICLE AND STANDARD ELEMENTS
        (SPLIT SPH AND STANDARD ELEMENTS)
      REBALANCE TYPE = AUTO_PERIODIC_ONLY|INITIAL_AND_AUTO_PERIODIC|
                       INITIAL_AND_PERIODIC|INITIAL_ONLY|PERIODIC_ONLY
      REBALANCE STRATEGY = SIMPLE|MULTICRITERIA(SIMPLE)
      REBALANCE STEP INTERVAL = <integer>step_interval(infinity)
      LOAD RATIO THRESHOLD = <real>load_ratio(1.05)
      COMMUNICATION RATIO THRESHOLD = <real>communication_ratio(1.15)
      ZOLTAN PARAMETERS = <string>parameter_nam
      DELETE DEACTIVATED ELEMENTS AFTER REBALANCE = ON|OFF(OFF)
      CLEAN UP GHOST ELEMENTS
      ACTIVE PERIODS = <string list>active_periods
      INACTIVE PERIODS = <string list>inactive_periods
    END [REBALANCE]

Initial decomposition of a mesh for parallel runs with Sierra/SM is done by a utility called `decomp`. The initial decomposition provided by `decomp` may become sub-optimal as the simulation evolves, especially for cases involving large deformations. Therefore, Presto supports a simple mesh-rebalancing capability that can be used to improve the parallel performance of some problems. When mesh rebalancing is invoked, the parallel decomposition is changed, and elements are moved among the processors to balance the computational load and minimize the processor-to-processor communication. Mesh rebalancing may be useful in the following circumstances:

- The mesh decomposition produced by `decomp` for sphere elements (i.e., SPH) can be poor. It is recommended that an initial mesh rebalance be done for all problems involving sphere elements.

- If a problem experiences very large deformations, periodic rebalancing may be helpful. In contact or SPH problems, communication is performed between physically nearby contact surfaces or SPH particles. To maintain optimum performance, it is helpful to have neighboring particles located on the same processors. Periodic mesh rebalancing can ensure that neighboring entities tend to remain on the same processor during large mesh deformations.

The `REBALANCE` command block is placed in the Presto region scope. The mesh rebalancing in Presto uses a mesh balancing library called Zoltan (Reference [[1]]). Zoltan performs the actual rebalancing. By default, Presto creates a Zoltan object with a default set of parameters. However, a Zoltan object with a customized set of parameters can be created and referenced from the `REBALANCE` command block.

### 6.8.1.1. Rebalance Command Lines

    ELEMENT GROUPING TYPE = SPLIT SPH AND STANDARD ELEMENTS|
      UNIFORM UNIFIED| SPLIT SPH AND DEM AND STANDARD ELEMENTS|
      SPLIT DEM AND STANDARD ELEMENTS |
      SPLIT PARTICLE AND STANDARD ELEMENTS
      (SPLIT SPH AND STANDARD ELEMENTS)
    REBALANCE TYPE = AUTO_PERIODIC_ONLY|INITIAL_AND_AUTO_PERIODIC|
                     INITIAL_AND_PERIODIC|INITIAL_ONLY|PERIODIC_ONLY
    REBALANCE STEP INTERVAL = <integer>step_interval(infinity)
    LOAD RATIO THRESHOLD = <real>load_ratio(1.05)
    COMMUNICATION RATIO THRESHOLD = <real>ratio(1.15)

The above command lines control how and when the rebalancing is done.

The `ELEMENT GROUPING TYPE` command line specifies how the elements are grouped for rebalancing. If set to `SPLIT SPH AND STANDARD ELEMENTS`, which is the default behavior, this command causes SPH and standard elements to be split into separate groups. If it is set to the `UNIFORM UNIFIED` option, all elements can be grouped together. This command also has options that relate to `DEM` an experimental element formulation. These `DEM` options are not recommended for general use.

The `REBALANCE TYPE` command line is used to specify the type of rebalance that is performed. If the `INITIAL_ONLY` option is used, the mesh will be rebalanced at time zero before any calculations occur. This option should be used if the initial mesh decomposition passed to Presto is poor. If the `PERIODIC_ONLY` option is used, the mesh will be rebalanced every `step_interval` steps, where `step_interval` is the parameter specified by the `REBALANCE STEP INTERVAL` command line. If the `AUTO_PERIODIC_ONLY` option is used, the mesh will be rebalanced every `step_interval` steps, when the communication ratio reaches a critical value, or when the load ratio reaches a critical value. All other values of `REBALANCE TYPE` are a combination of the above three options.

The `REBALANCE STRATEGY` command line specifies which algorithm to use to perform the rebalance. `SIMPLE` is the legacy approach which simply distributes the elements or load evenly across all of the processors used. The `MULTICRITERIA` option dynamically manages different physical models and their relative cost. For example, the `MULTICRITERIA` option accounts for contact boundaries and the relative cost of different element topologies (such as shell and solid elements). Note that not all element types have been assigned weights for multicriteria rebalance.

The communication ratio, currently defined only for SPH problems, is a measure of how much communication is required in the current mesh decomposition versus an estimate of the amount of communication with an optimal decomposition. Mesh rebalancing is expensive, so rebalancing should be done rarely. The `COMMUNICATION RATIO THRESHOLD` command line specifies the value of the communication ratio that triggers a rebalance. Setting this to a value between 1.25 and 1.5 is usually optimal.

The load ratio is the ratio of the current load (maximum number of elements on a processor divided by the average number of elements per processor). The `LOAD RATIO THRESHOLD` command specifies that a rebalance should occur when the load ratio exceeds the specified threshold. A specified value of 1.5 would trigger a rebalance when the load ratio is 50% higher than could be optimally obtained.

The `DELETE DEACTIVATED ELEMENTS AFTER REBALANCE` command line can be used to remove any inactive elements before performing the mesh rebalance. If a large number of elements have been deactivated due to element death, or due to use of time step sub-cycling, this may lead to a better load balance. With this option rebalance will get an optimal decomposition considering only those elements that are currently alive and active and remove any other elements permanently from the data structure.

The `CLEAN UP GHOST ELEMENTS` command line can be used to delete ghosts and trigger re-ghosting. This is beneficial with particles and other things that create lots of ghosting that can change as the problem deforms.

The `ACTIVE PERIODS` AND `INACTIVE PERIODS` commands can turn rebalancing on or off by time period. See Section 2.6 for more information regarding these commands.

### 6.8.1.2. Zoltan Command Line

The command line

    ZOLTAN PARAMETERS = <string>parameter_name

references a `ZOLTAN PARAMETERS` command block named `parameter_name`. Various parameters for Zoltan can be set in the `ZOLTAN PARAMETERS` command block. If the `ZOLTAN PARAMETERS` command line is not provided, a default set of parameters is used. The default parameter command block is shown as follows:

    BEGIN ZOLTAN PARAMETERS
      LOAD BALANCING METHOD = recursive coordinate bisection
        # string parameter
      OVER ALLOCATE MEMORY  = 1.5 # real parameter
      REUSE CUTS = true # string parameter
      ALGORITHM DEBUG LEVEL = 0 # integer parameter
      CHECK GEOMETRY = true # string parameter
      ZOLTAN DEBUG LEVEL = 0 # integer parameter
    END ZOLTAN PARAMETERS

See Section 6.8.2 for a discussion of the `ZOLTAN PARAMETERS` command block. Section 6.8.2 lists the command lines that can be used to set Zoltan parameters.

## 6.8.2. Zoltan Parameters

    BEGIN ZOLTAN PARAMETERS
      LOAD BALANCING METHOD  = <string>recursive coordinate bisection|
        recursive inertial bisection|Hilbert space filling curve
      DETERMINISTIC DECOMPOSITION = <string>false|true
      IMBALANCE TOLERANCE = <real>imb_tol
      OVER ALLOCATE MEMORY = <real>over_all_mem
      REUSE CUTS = <string>false|true
      ALGORITHM DEBUG LEVEL = <integer>alg_level
        #  0<=(alg_level)<=3
      CHECK GEOMETRY = <string>false|true
      KEEP CUTS = <string>false|true
      LOCK RCB DIRECTIONS = <string>false|true
      SET RCB DIRECTIONS = <string>do not order cuts|xyz|xzy|
        yzx|yxz|zxy|zyx
      RECTILINEAR RCB BLOCKS = <string>false|true
      RENUMBER PARTITIONS = <string>false|true
      ZOLTAN DEBUG LEVEL = <integer>zoltan_level
        #  0<=(zoltan_level)<=10
      DEBUG PROCESSOR NUMBER = <integer>proc  #  1<=proc
      TIMER = <string> wall|cpu
      DEBUG MEMORY = <integer>dbg_mem  #  0<=(dbg_mem)<=3
    END [ZOLTAN PARAMETERS]

The `ZOLTAN PARAMETERS` command block is used to set parameters for Zoltan (see Reference [[1]]), a program that can be used for mesh rebalancing in Presto. The `ZOLTAN PARAMETERS` command block is used in association with the `REBALANCE` command block. A `REBALANCE` command block may reference a `ZOLTAN PARAMETERS` command block via the name, `parameter_name`, for the parameter command block. Reference Section 6.8.1 regarding the use of the `ZOLTAN PARAMETERS` command block for mesh rebalancing in Presto.

Setting the parameters for Zoltan involves some understanding of how Zoltan works. Consult Reference [[1]] for a discussion of the parameters that can be set by the various command lines in the `ZOLTAN PARAMETERS` command block. Note that some of the command lines in this command block have comments that provide additional information about the parameters. The “#” symbol precedes a comment.

In the above command block, \(=\) and `IS` are the allowed delimiters, which is different from the usual Presto convention of \(=\), `IS`, and `ARE`. Note that the `ZOLTAN PARAMETERS` command block should be specified in the SIERRA scope when it is referenced from the `ZOLTAN PARAMETERS` command line in the `REBALANCE` command block. When the default set of parameters is used for a Zoltan object, the `ZOLTAN PARAMETERS` command block need not be included in the input file.

[1] (1,2,3)

K. Devine, E. Boman, R. Heapby, B. Hendrickson, and C. Vaughan. Zoltan data management service for parallel dynamic applications. Computing in Science and Engineering, 4:90–97, 2002. URL: http://www.cs.sandia.gov/zoltan, doi:10.1109/5992.988653.