6.5. Troubleshooting
Sometimes an Aria job will fail. This can be for a number of reasons, including invalid syntax in the input file, an ill-posed or non-physical problem setup, insufficient mesh resolution, computing hardware issues, or other reasons. When this happens, you will need to look for clues in the log file, the stdout stream (or slurm file), and any results output files (Exodus or Heartbeat). The following sections address some of the common failure modes you might encounter.
6.5.1. Invalid Input Syntax
If you use something in the input file that is not valid Aria syntax, you will get an error in the log file. If the problem is invalid syntax, the error will be at the point in the input file where the invalid syntax was, and will suggest alternative syntax. For example, if we use the wrong syntax for a constant thermal conductivity (using tk instead of value or k for the constant)
BEGIN ARIA MATERIAL Kryptonite
Thermal Conductivity = constant tk=1.0
The first signal we get that something is wrong is in the stdout stream (or the slurm file if launched on an HPC), but it does not tell us what the specific error is.
$ aria -i demo.i
exception on all processors:
Execution terminated due to errors
*** SIERRA ABORT on P0 ***
*** check the log file for more information ***
Looking in the log file, at the end of the file we would see
There was 1 error encountered during parse
There were no warnings encountered during parse
Execution terminated due to errors
SIERRA execution failed during parse with the following exception:
Execution terminated due to errors
This does not tell us what the error is by itself either, but since it says it was a parse error we should search for the keyword “error” in the section of the log file where it repeats the input file. Then we find
BEGIN SIERRA myJob
BEGIN ARIA MATERIAL Kryptonite
Thermal Conductivity = constant tk=1.0
demo.i:7: Error: No matching command line found for 'Thermal Conductivity = constant tk=1.0'
demo.i:7: The following command lines with matching keywords are defined:
demo.i:7: Thermal Conductivity [ {Of|Species|Subindex} <speciesname: string> ] = Constant {K|Value} = <k: real>
demo.i:7:
heat conduction = basic
The error message tells us which line was invalid, and shows the syntax for the best match. In this case, the model we used (constant) is a valid model, so it only prints the syntax for that model. If we had used an invalid model (e.g. constnt), the list of possibilities would include all the valid models for thermal conductivity (shown below with a truncated list)
BEGIN ARIA MATERIAL Kryptonite
Thermal Conductivity = constnt k=1.0
demo.i:7: Error: No matching command line found for 'Thermal Conductivity = constnt k=1.0'
demo.i:7: The following command lines with matching keywords are defined:
demo.i:7: Thermal Conductivity [ {Of|Species|Subindex} <speciesname: string> ] = T_Exponent K_Ref = <k_ref:
demo.i:7: real> T_Ref = <t_ref: real> N = <n: real>
demo.i:7: Thermal Conductivity [ {Of|Species|Subindex} <speciesname: string> ] = Volume_Average
demo.i:7: Thermal Conductivity [ {Of|Species|Subindex} <speciesname: string> ] = Summed Contributions =
demo.i:7: <contributions: string>
...
6.5.2. Deprecated Syntax
Most Aria problems use stable syntax that does not change from one release to the next, however sometimes features need to be removed for maintainability and sustainability of the codebase. This is done using a deprecation cycle to allow for time to migrate input file syntax.
Aria uses a 6 to 12 month deprecation cycle (or longer for larger feature deprecations). When a deprecation is introduced, it will result in a warning immediately, which shows up both in the log file and in the stderr/slurm stream. For example, if you used a deprecated post-processor in Aria version 5.10 you would see the following warning in the stderr/slurm output
WARNING: Deprecated feature removed in Version 5.11 detected.
The 'POST PROCESS FLUX' solution options command is deprecated in favor of the new postprocessing syntax. Replace
BEGIN SOLUTION OPTIONS
post process FLUX heat_conduction on fluid_interface as HeatFlux
END SOLUTION OPTIONS
with
postprocess expression_flux of expression heat_conduction on fluid_interface as HeatFlux
for a direct replacement. See Sections 21.5 of the user manual for more information.
Please email sierra-help@sandia.gov if you have issues converting post process commands.
The same warning would also show up in the log file
demo.i:82: Warning: Deprecated feature removed in Version 5.11 detected.
demo.i:82: The 'POST PROCESS FLUX' solution options command is deprecated in favor of the new postprocessing
demo.i:82: syntax. Replace
demo.i:82: BEGIN SOLUTION OPTIONS
demo.i:82: post process FLUX heat_conduction on fluid_interface as HeatFlux
demo.i:82: END SOLUTION OPTIONS
demo.i:82: with
demo.i:82: postprocess expression_flux of expression heat_conduction on fluid_interface as HeatFlux
demo.i:82: for a direct replacement. See Sections 21.5 of the user manual for more information.
demo.i:82: Please email sierra-help@sandia.gov if you have issues converting post process commands.
demo.i:82: in line command POST PROCESS FLUX
Deprecation version numbers are always odd-numbered (non-release). When the 5.10 version is released, the version of sierra/daily is set to 5.11. In this example, this means that using the deprecated post-processor will be an error if you run sierra/daily during the 5.11 release cycle, and in the next release (5.12) it will be an error. Pay attention to these warnings and update your inputs in a timely manner to avoid getting errors at future releases.
6.5.3. Non-Physical Temperatures
The conservation of energy equation in its continuous form (and in the absence of source terms) satisfies a maximum principle. That is, for a steady-state problem the maximum and minimum temperatures in the domain must occur on the boundary of the domain, and for a transient problem on either the boundary or the initial condition. However, the discretized form of the energy equation does not necessarily satisfy a discrete version of the maximum principle (DMP). In particular for the standard Galerkin finite element method used in Aria there are conditions on the mesh quality that must be met for the diffusion operator to satisfy a DMP.
The article “Don’t suppress the wiggles—They’re telling you something!” can provide useful context on the nature of oscillations in finite element solutions and provides valuable background information as to why they occur and what they mean for your solution.
In general, if you encounter non-physical high or low temperatures the troubleshooting steps you should start with are:
Examine the mesh in the region of spurious temperature to ensure you have enough resolution in the direction of heat flow. Pay particular attention to thin regions and make sure they are more than 1 element thick. Also note that there can be issues on edge/corner elements that have multiple different boundary conditions applied to them, particularly with tetrahedral elements.
Try using first order time integration, lumped_mass, and lumped_flux boundary conditions (although this will increase discretization error)
Enable the flux limiter option if using a tetrahedral mesh (described below).
For linear tetrahedral elements the DMP condition is that there are no obtuse dihedral angles between faces of elements in the mesh [44]. Anisotropic thermal conductivity can also contribute to this effect [45]. Additionally, the consistent mass term can also lead to DMP violations in transient problems [46]. In practice this can lead to non-physical temperatures appearing in the solutions generated by Aria, most commonly when large heat fluxes are applied to an initially cold domain with a low thermal conductivity.
If non-physical temperature solutions are observed Aria supports an option to apply
a nonlinear flux correction based on the work of Kuzmin et al. that restores a
DMP on arbitrary meshes with arbitrary material properties
[46, 47]. This option may be
activated by using the APPLY FLUX LIMITER STABILIZATION line command.
For transient problems it is also essential to use the
LUMPED_MASS form of the time derivative term. At present, this option is only intended
for use with linear elements, not second order 10 node tetrahedral elements or 27 node
hexahedral elements. Source terms also do not have any limiters applied to them at
present, as a result they may still cause non-physical temperatures in some cases.
Additionally, applying the stabilization has several downsides so that we only recommend enabling it if problematic non-physical temperatures are observed without it. Solutions with the stabilization activated are more diffusive and have higher error than unstabilized solutions (though they do converge at the same order with mesh refinement). The stabilization operator is also nonlinear and can adversely affect the convergence of the Newton-Raphson iteration within each time step. Combined with the cost of calculating the stabilization terms this can have noticeable impact on simulation runtime.
6.5.4. Repeated Time Step Failure
If your problem fails a timestep, Aria will decrease the step size and attempt the solve(s) again. If this keeps happening, eventually Aria will fail when the time step reaches the allowable minimum. Some strategies to try if you encounter this type of failure are:
Ensure that your problem is mathematically well-posed and that the linear solves are completing with a sufficiently low final residual (they should be lower than your nonlinear residual target).
Increase the number of nonlinear iterations if the problem was converging but not reaching your target residual before hitting the iteration limit.
Change the nonlinear solution strategy from
NewtontoLine_searchto enable an adaptive under-relaxation that attempts to ensure that each nonlinear step always results in a reduction in the nonlinear residual. This will increase the cost of each nonlinear iteration.
6.5.5. Linear Solver Failure
If the linear solver fails to solve your linear system there will be an error code reported in the log file. The Solver Error Codes section outlines the meaning behind these error codes. In general, if you see error codes you should re-examine your choice of linear solver or check whether the problem as-posed is non-singular. The Solver Selection Guidelines section has suggestions for how to incrementally adjust the solver settings for harder-to-solve problems.
6.5.6. Out of Memory
When running memory intensive computations like calculating view factors on memory limited HPC platforms, your simulation will be killed if the compute node runs out of memory. This can also happen if you run a conventional problem with too many elements per core. Unfortunately there is no hard guideline as to the maximum problem size since it depends on what options are used in the simulation and varies between HPC platforms, but without enclosure radiation it should be able to support several million elements per core before running into memory issues.
When a simulation on the HPC runs out of memory, the user who launched it will typically receive an automated email from the HPC system notifying them that their job ran out of memory, although this notification may not be sent immediately. If your jobs are running out of memory you generally have a few options:
Run the job on a larger number of nodes or cores
Use fewer cores per node than the total on an HPC platform (e.g. request twice as many nodes but use half as many cores per node)
Run the job on a different system that has more memory per core available
Since most memory-limited simulations in Aria are due to view factor calculations for enclosure radiation, the Chapparal log file also includes estimates on memory use to help you size your job submission appropriately.