6.8. Mesh Rebalancing

Mesh rebalancing is a feature in Presto that may improve the efficiency of an analysis. Two command blocks can be used to control mesh rebalancing: REBALANCE and ZOLTAN PARAMETERS. The REBALANCE command block is required; the ZOLTAN PARAMETERS command block is optional. Sections Section 6.8.1 and Section 6.8.2 describe these command blocks.

6.8.1. Rebalance

BEGIN REBALANCE
  ELEMENT GROUPING TYPE = SPLIT SPH AND STANDARD ELEMENTS|
    UNIFORM UNIFIED| SPLIT SPH AND DEM AND STANDARD ELEMENTS|
    SPLIT DEM AND STANDARD ELEMENTS |
    SPLIT PARTICLE AND STANDARD ELEMENTS
    (SPLIT SPH AND STANDARD ELEMENTS)
  REBALANCE TYPE = AUTO_PERIODIC_ONLY|INITIAL_AND_AUTO_PERIODIC|
                   INITIAL_AND_PERIODIC|INITIAL_ONLY|PERIODIC_ONLY
  REBALANCE STRATEGY = SIMPLE|MULTICRITERIA(SIMPLE)
  REBALANCE STEP INTERVAL = <integer>step_interval(infinity)
  LOAD RATIO THRESHOLD = <real>load_ratio(1.05)
  COMMUNICATION RATIO THRESHOLD = <real>communication_ratio(1.15)
  ZOLTAN PARAMETERS = <string>parameter_nam
  DELETE DEACTIVATED ELEMENTS AFTER REBALANCE = ON|OFF(OFF)
  CLEAN UP GHOST ELEMENTS
  ACTIVE PERIODS = <string list>active_periods
  INACTIVE PERIODS = <string list>inactive_periods
END [REBALANCE]

Initial decomposition of a mesh for parallel runs with Sierra/SM is done by a utility called decomp. The initial decomposition provided by decomp may become sub-optimal as the simulation evolves, especially for cases involving large deformations. Therefore, Presto supports a simple mesh-rebalancing capability that can be used to improve the parallel performance of some problems. When mesh rebalancing is invoked, the parallel decomposition is changed, and elements are moved among the processors to balance the computational load and minimize the processor-to-processor communication. Mesh rebalancing may be useful in the following circumstances:

• The mesh decomposition produced by decomp for sphere elements (i.e., SPH) can be poor. It is recommended that an initial mesh rebalance be done for all problems involving sphere elements.

• If a problem experiences very large deformations, periodic rebalancing may be helpful. In contact or SPH problems, communication is performed between physically nearby contact surfaces or SPH particles. To maintain optimum performance, it is helpful to have neighboring particles located on the same processors. Periodic mesh rebalancing can ensure that neighboring entities tend to remain on the same processor during large mesh deformations.

The REBALANCE command block is placed in the Presto region scope. The mesh rebalancing in Presto uses a mesh balancing library called Zoltan (Reference [[1]]). Zoltan performs the actual rebalancing. By default, Presto creates a Zoltan object with a default set of parameters. However, a Zoltan object with a customized set of parameters can be created and referenced from the REBALANCE command block.

6.8.1.1. Rebalance Command Lines

ELEMENT GROUPING TYPE = SPLIT SPH AND STANDARD ELEMENTS|
  UNIFORM UNIFIED| SPLIT SPH AND DEM AND STANDARD ELEMENTS|
  SPLIT DEM AND STANDARD ELEMENTS |
  SPLIT PARTICLE AND STANDARD ELEMENTS
  (SPLIT SPH AND STANDARD ELEMENTS)
REBALANCE TYPE = AUTO_PERIODIC_ONLY|INITIAL_AND_AUTO_PERIODIC|
                 INITIAL_AND_PERIODIC|INITIAL_ONLY|PERIODIC_ONLY
REBALANCE STEP INTERVAL = <integer>step_interval(infinity)
LOAD RATIO THRESHOLD = <real>load_ratio(1.05)
COMMUNICATION RATIO THRESHOLD = <real>ratio(1.15)

The above command lines control how and when the rebalancing is done.

The ELEMENT GROUPING TYPE command line specifies how the elements are grouped for rebalancing. If set to SPLIT SPH AND STANDARD ELEMENTS, which is the default behavior, this command causes SPH and standard elements to be split into separate groups. If it is set to the UNIFORM UNIFIED option, all elements can be grouped together. This command also has options that relate to DEM an experimental element formulation. These DEM options are not recommended for general use.

The REBALANCE TYPE command line is used to specify the type of rebalance that is performed. If the INITIAL_ONLY option is used, the mesh will be rebalanced at time zero before any calculations occur. This option should be used if the initial mesh decomposition passed to Presto is poor. If the PERIODIC_ONLY option is used, the mesh will be rebalanced every step_interval steps, where step_interval is the parameter specified by the REBALANCE STEP INTERVAL command line. If the AUTO_PERIODIC_ONLY option is used, the mesh will be rebalanced every step_interval steps, when the communication ratio reaches a critical value, or when the load ratio reaches a critical value. All other values of REBALANCE TYPE are a combination of the above three options.

The REBALANCE STRATEGY command line specifies which algorithm to use to perform the rebalance. SIMPLE is the legacy approach which simply distributes the elements or load evenly across all of the processors used. The MULTICRITERIA option dynamically manages different physical models and their relative cost. For example, the MULTICRITERIA option accounts for contact boundaries and the relative cost of different element topologies (such as shell and solid elements). Note that not all element types have been assigned weights for multicriteria rebalance.

The communication ratio, currently defined only for SPH problems, is a measure of how much communication is required in the current mesh decomposition versus an estimate of the amount of communication with an optimal decomposition. Mesh rebalancing is expensive, so rebalancing should be done rarely. The COMMUNICATION RATIO THRESHOLD command line specifies the value of the communication ratio that triggers a rebalance. Setting this to a value between 1.25 and 1.5 is usually optimal.

The load ratio is the ratio of the current load (maximum number of elements on a processor divided by the average number of elements per processor). The LOAD RATIO THRESHOLD command specifies that a rebalance should occur when the load ratio exceeds the specified threshold. A specified value of 1.5 would trigger a rebalance when the load ratio is 50% higher than could be optimally obtained.

The DELETE DEACTIVATED ELEMENTS AFTER REBALANCE command line can be used to remove any inactive elements before performing the mesh rebalance. If a large number of elements have been deactivated due to element death, or due to use of time step sub-cycling, this may lead to a better load balance. With this option rebalance will get an optimal decomposition considering only those elements that are currently alive and active and remove any other elements permanently from the data structure.

The CLEAN UP GHOST ELEMENTS command line can be used to delete ghosts and trigger re-ghosting. This is beneficial with particles and other things that create lots of ghosting that can change as the problem deforms.

The ACTIVE PERIODS AND INACTIVE PERIODS commands can turn rebalancing on or off by time period. See Section 2.6 for more information regarding these commands.

6.8.1.2. Zoltan Command Line

The command line

ZOLTAN PARAMETERS = <string>parameter_name

references a ZOLTAN PARAMETERS command block named parameter_name. Various parameters for Zoltan can be set in the ZOLTAN PARAMETERS command block. If the ZOLTAN PARAMETERS command line is not provided, a default set of parameters is used. The default parameter command block is shown as follows:

BEGIN ZOLTAN PARAMETERS
  LOAD BALANCING METHOD = recursive coordinate bisection
    # string parameter
  OVER ALLOCATE MEMORY  = 1.5 # real parameter
  REUSE CUTS = true # string parameter
  ALGORITHM DEBUG LEVEL = 0 # integer parameter
  CHECK GEOMETRY = true # string parameter
  ZOLTAN DEBUG LEVEL = 0 # integer parameter
END ZOLTAN PARAMETERS

See Section 6.8.2 for a discussion of the ZOLTAN PARAMETERS command block. Section 6.8.2 lists the command lines that can be used to set Zoltan parameters.

6.8.2. Zoltan Parameters

BEGIN ZOLTAN PARAMETERS
  LOAD BALANCING METHOD  = <string>recursive coordinate bisection|
    recursive inertial bisection|Hilbert space filling curve
  DETERMINISTIC DECOMPOSITION = <string>false|true
  IMBALANCE TOLERANCE = <real>imb_tol
  OVER ALLOCATE MEMORY = <real>over_all_mem
  REUSE CUTS = <string>false|true
  ALGORITHM DEBUG LEVEL = <integer>alg_level
    #  0<=(alg_level)<=3
  CHECK GEOMETRY = <string>false|true
  KEEP CUTS = <string>false|true
  LOCK RCB DIRECTIONS = <string>false|true
  SET RCB DIRECTIONS = <string>do not order cuts|xyz|xzy|
    yzx|yxz|zxy|zyx
  RECTILINEAR RCB BLOCKS = <string>false|true
  RENUMBER PARTITIONS = <string>false|true
  ZOLTAN DEBUG LEVEL = <integer>zoltan_level
    #  0<=(zoltan_level)<=10
  DEBUG PROCESSOR NUMBER = <integer>proc  #  1<=proc
  TIMER = <string> wall|cpu
  DEBUG MEMORY = <integer>dbg_mem  #  0<=(dbg_mem)<=3
END [ZOLTAN PARAMETERS]

The ZOLTAN PARAMETERS command block is used to set parameters for Zoltan (see Reference [[1]]), a program that can be used for mesh rebalancing in Presto. The ZOLTAN PARAMETERS command block is used in association with the REBALANCE command block. A REBALANCE command block may reference a ZOLTAN PARAMETERS command block via the name, parameter_name, for the parameter command block. Reference Section 6.8.1 regarding the use of the ZOLTAN PARAMETERS command block for mesh rebalancing in Presto.

Setting the parameters for Zoltan involves some understanding of how Zoltan works. Consult Reference [[1]] for a discussion of the parameters that can be set by the various command lines in the ZOLTAN PARAMETERS command block. Note that some of the command lines in this command block have comments that provide additional information about the parameters. The “#” symbol precedes a comment.

In the above command block, \(=\) and IS are the allowed delimiters, which is different from the usual Presto convention of \(=\), IS, and ARE. Note that the ZOLTAN PARAMETERS command block should be specified in the SIERRA scope when it is referenced from the ZOLTAN PARAMETERS command line in the REBALANCE command block. When the default set of parameters is used for a Zoltan object, the ZOLTAN PARAMETERS command block need not be included in the input file.

---

[1] (1,2,3)

K. Devine, E. Boman, R. Heapby, B. Hendrickson, and C. Vaughan. Zoltan data management service for parallel dynamic applications. Computing in Science and Engineering, 4:90–97, 2002. URL: http://www.cs.sandia.gov/zoltan, doi:10.1109/5992.988653.