Explicit only

6.8. Mesh Rebalancing

Mesh rebalancing is a feature in Sierra/SM Explicit that may improve the efficiency of an analysis. The command block to control mesh rebalancing is REBALANCE.

BEGIN REBALANCE
  ELEMENT GROUPING TYPE = SPLIT SPH AND STANDARD ELEMENTS|
    UNIFORM UNIFIED| SPLIT SPH AND DEM AND STANDARD ELEMENTS|
    SPLIT DEM AND STANDARD ELEMENTS |
    SPLIT PARTICLE AND STANDARD ELEMENTS
    (SPLIT SPH AND STANDARD ELEMENTS)
  REBALANCE TYPE = AUTO_PERIODIC_ONLY|INITIAL_AND_AUTO_PERIODIC|
                   INITIAL_AND_PERIODIC|INITIAL_ONLY|PERIODIC_ONLY
  REBALANCE STRATEGY = SIMPLE|MULTICRITERIA(SIMPLE)
  REBALANCE STEP INTERVAL = <integer>step_interval(infinity)
  LOAD RATIO THRESHOLD = <real>load_ratio(1.05)
  COMMUNICATION RATIO THRESHOLD = <real>communication_ratio(1.15)
  DELETE DEACTIVATED ELEMENTS AFTER REBALANCE = ON|OFF(OFF)
  CLEAN UP GHOST ELEMENTS
  ACTIVE PERIODS = <string list>active_periods
  INACTIVE PERIODS = <string list>inactive_periods
END [REBALANCE]

Initial decomposition of a mesh for parallel runs with Sierra/SM is done by a utility called decomp. The initial decomposition provided by decomp may become sub-optimal as the simulation evolves, especially for cases involving large deformations. Therefore, Sierra/SM Explicit supports a simple mesh-rebalancing capability that can be used to improve the parallel performance of some problems. When mesh rebalancing is invoked, the parallel decomposition is changed, and elements are moved among the processors to balance the computational load and minimize the processor-to-processor communication. Mesh rebalancing may be useful in the following circumstances:

The mesh decomposition produced by decomp for sphere elements (i.e., SPH) can be poor. It is recommended that an initial mesh rebalance be done for all problems involving sphere elements.
If a problem experiences very large deformations, periodic rebalancing may be helpful. In contact or SPH problems, communication is performed between physically nearby contact surfaces or SPH particles. To maintain optimum performance, it is helpful to have neighboring particles located on the same processors. Periodic mesh rebalancing can ensure that neighboring entities tend to remain on the same processor during large mesh deformations.

The REBALANCE command block is placed in the Presto region scope.

ELEMENT GROUPING TYPE = SPLIT SPH AND STANDARD ELEMENTS|
  UNIFORM UNIFIED| SPLIT SPH AND DEM AND STANDARD ELEMENTS|
  SPLIT DEM AND STANDARD ELEMENTS |
  SPLIT PARTICLE AND STANDARD ELEMENTS
  (SPLIT SPH AND STANDARD ELEMENTS)
REBALANCE TYPE = AUTO_PERIODIC_ONLY|INITIAL_AND_AUTO_PERIODIC|
                 INITIAL_AND_PERIODIC|INITIAL_ONLY|PERIODIC_ONLY
REBALANCE STEP INTERVAL = <integer>step_interval(infinity)
LOAD RATIO THRESHOLD = <real>load_ratio(1.05)
COMMUNICATION RATIO THRESHOLD = <real>ratio(1.15)

The above command lines control how and when the rebalancing is done.

The ELEMENT GROUPING TYPE command line specifies how the elements are grouped for rebalancing. If set to SPLIT SPH AND STANDARD ELEMENTS, which is the default behavior, this command causes SPH and standard elements to be split into separate groups. If it is set to the UNIFORM UNIFIED option, all elements can be grouped together. This command also has options that relate to DEM an experimental element formulation. These DEM options are not recommended for general use.

The REBALANCE TYPE command line is used to specify the type of rebalance that is performed. If the INITIAL_ONLY option is used, the mesh will be rebalanced at time zero before any calculations occur. This option should be used if the initial mesh decomposition passed to Sierra/SM Explicit is poor. If the PERIODIC_ONLY option is used, the mesh will be rebalanced every step_interval steps, where step_interval is the parameter specified by the REBALANCE STEP INTERVAL command line. If the AUTO_PERIODIC_ONLY option is used, the mesh will be rebalanced every step_interval steps, when the communication ratio reaches a critical value, or when the load ratio reaches a critical value. All other values of REBALANCE TYPE are a combination of the above three options.

The REBALANCE STRATEGY command line specifies which algorithm to use to perform the rebalance. SIMPLE is the legacy approach which simply distributes the elements or load evenly across all of the processors used. The MULTICRITERIA option dynamically manages different physical models and their relative cost. For example, the MULTICRITERIA option accounts for contact boundaries and the relative cost of different element topologies (such as shell and solid elements). Note that not all element types have been assigned weights for multicriteria rebalance.

The communication ratio, currently defined only for SPH problems, is a measure of how much communication is required in the current mesh decomposition versus an estimate of the amount of communication with an optimal decomposition. Mesh rebalancing is expensive, so rebalancing should be done rarely. The COMMUNICATION RATIO THRESHOLD command line specifies the value of the communication ratio that triggers a rebalance. Setting this to a value between 1.25 and 1.5 is usually optimal.

The load ratio is the ratio of the current load (maximum number of elements on a processor divided by the average number of elements per processor). The LOAD RATIO THRESHOLD command specifies that a rebalance should occur when the load ratio exceeds the specified threshold. A specified value of 1.5 would trigger a rebalance when the load ratio is 50% higher than could be optimally obtained.

The DELETE DEACTIVATED ELEMENTS AFTER REBALANCE command line can be used to remove any inactive elements before performing the mesh rebalance. If a large number of elements have been deactivated due to element death, or due to use of time step sub-cycling, this may lead to a better load balance. With this option rebalance will get an optimal decomposition considering only those elements that are currently alive and active and remove any other elements permanently from the data structure.

The CLEAN UP GHOST ELEMENTS command line can be used to delete ghosts and trigger re-ghosting. This is beneficial with particles and other things that create lots of ghosting that can change as the problem deforms.

The ACTIVE PERIODS AND INACTIVE PERIODS commands can turn rebalancing on or off by time period. See Section 2.6 for more information regarding these commands.