6.1. Running Aria
6.1.1. Loading the Sierra Module
Once you have finished setting up an input file, you are ready to run Aria. From a CEE or HPC UNIX environment at Sandia, you can load the sierra module to access the latest release of Aria.
$ module load sierra
This will load the current release of Sierra. To load other versions of Sierra, you can use the following modules
module load sierra/x.x- Load versionx.x(e.g.module load sierra/5.10)
module load sierra/sprint- Load the latest sprint release (every 3-week release)
module load sierra/daily- Load the daily build of Sierra.
Warning
Using the sierra/daily module exposes you to potential bugs and instabilities since it is actively developed. If the nightly Sierra build process fails, the sierra/daily executable may not exist, or may be much older than expected.
To see a list of the available Sierra versions (and other useful modules available like apps/anaconda3 or apps/matlab) you can use
$ module avail
6.1.2. Running Aria Locally
To run a job on a non-queued system (e.g. a CEE blade or a cee compute machine) you can call launch or mpirun. For example, to run a job on 4 processors, you would use
$ module load sierra
$ launch -n 4 aria -i demo.i
The launch command is usually equivalent to using mpirun for local execution, but handles setting required MPI flags when running on some HPC systems.
$ module load sierra
$ mpirun -np 4 aria -i demo.i
6.1.3. Using the Sierra Script
The sierra script included in the sierra modules provides some additional functionality for launching sierra jobs. Its use is similar to the launch and mpirun commands.
$ sierra --np 4 aria -i demo.i
By default, the sierra script will perform extra steps that are not necessary for running Aria. These include:
Reading the input file to find the mesh file and running
decompon it beforehand.Reading the input file to find the output files and running
epuon them after the simulation is done.
Aria will automatically decompose your mesh, which is usually faster than manually running decomp, and most visualization tools can view decomposed output files so running epu to combine them into a single file is usually not needed either. To use the sierra script without invoking those steps, add the --run option.
$ sierra --run --np 4 aria -i demo.i
The sierra script can also be used to build user plugins using the --make option. See the sections on user plugins and user subroutines for more information.
$ sierra --make aria -i usersub_demo.i
6.1.4. Running Aria on an HPC
The HPC systems at Sandia use slurm to schedule jobs. To run a job on an HPC you need to submit it to the slurm system along with some additional information (listed below) and the system will put your job in the queue and run at some point later in the future.
The number of compute nodes to run on.
The number of cores to use per node.
The wall-clock duration of the job (slurm will kill the job if it’s not done by this time limit).
A WCID for the job based on the project funding it. The WCID is used for tracking purposes and also determines the job priority. Use the WC Tool web site to check your WCIDs or get a new one.
Which queue to submit to (most HPCs have “batch”, “short”, and “long” - “batch” is the standard).
Refer to the HPC homepage for details about queue limits, core counts per node, and other useful HPC information for the machine you intend to run on.
If you are using SAW to submit your jobs, it can handle collecting the required information and submitting the job to the queue. If you do not use SAW, you can submit your jobs manually using either a batch script or the sierra script.
For simple job submissions you can use the sierra script directly after logging in to the HPC you want to run on. For example, to run Aria on 360 processors for up to 24 hours you would log in to the HPC and run the following command.
$ sierra --run --np 360 aria -i demo.i --account WCID --queue-name batch --time-limit 24:00:00
For more complicated submissions, you may need to prepare a batch script to perform any custom pre-processing steps you need. There are example submission scripts for different platforms in /projects/samples at Sandia. An example script to run Aria is shown below.
#!/bin/bash
#SBATCH --nodes=10 # Number of nodes
#SBATCH --ntasks-per-node=36 # Number of cores per node
#SBATCH --time=24:00:00 # Wall clock time (HH:MM:SS)
#SBATCH --account=PUT_YOUR_WCID_HERE # WC ID
#SBATCH --job-name=test # Name of job
#SBATCH --partition=batch # partition/queue name: short or batch
nodes=$SLURM_JOB_NUM_NODES
cores=36
# do any pre-processing steps you need
mpiexec --bind-to core --npernode $cores --n $(($cores*$nodes)) aria -i demo.i
If you saved the above script as run_aria then you would submit it to the queue using
$ sbatch run_aria
You can check on the status of your queued jobs using squeue -u myusername. To see an estimate of when they
will start, use squeue -u myusername --start.
6.1.5. Running MPMD Jobs
The command to run in MPMD mode is different from what is used to run Aria normally. To run Aria cases you simply run a parallel job of Aria using something discussed above e.g.
$ mpirun -np 100 aria -i input.i
In this case one executable, Aria, is using 100 cores. With MPMD runs you are launching two separate MPI jobs with two different codes that can communicate. An example MPMD launch command would look like
$ mpirun -np 100 aria --mpmd_master --mpi_color 9999 \
-i input.i : -np 100 follower_app [follower_app args...]
The mpmd_master flag indicates that Aria will be controlling the execution (the “leader”) of the simulation
while the other application “follows” Aria. Aria can be run as follower by omitting this flag e.g.
$ mpirun -np 100 aria --mpi_color 9999 \
-i input.i : -np 100 leader_app [leader_app args...]
Note that the order of the two apps does not matter. The mpi_color argument specifies
an id to label the cores executing Aria with to
distinguish them from the cores executing the coupled application. This can be any integer,
and must be unique across the coupled apps.
There is no requirement that the two codes use the same number of cores, so depending on the mesh and computational costs you may choose a different allocation per code. For example, if your coupled app is very expensive, you may allocate more cores to it than Aria:
$ mpirun -np 50 aria --mpmd_master --mpi_color 9999 \
-i input.i : -np 150 other_app [other_app args...]
Special care must be taken when submitting MPMD jobs on the HPCs or any queued environment. By default, the two MPMD codes cannot share cores so to launch the case above on an HPC you would need to request an allocation of 200 cores. This is unnecessarily wasteful though if the coupled app would not be using its 150 cores while Aria runs, and Aria would not be using its 50 cores while the coupled app. To get around this, you must enable oversubscription. For couples where the two apps run sequentially (never executing at the same time) you can allow them to share resources. To get an allocation of 100 cores and use them all for both codes you must add additional mpi flags:
$ mpiexec --oversubscribe --bind-to core:overload-allowed -np 100 \
aria --mpmd_master --mpi_color 9999 -i input.i : \
--bind-to core:overload-allowed -np 100 other_app [other_app args...]
Keep in mind that the specific command to use can be platform dependent. A more complete example submission script on an HPC may look like
#!/bin/bash
#SBATCH --nodes=10
#SBATCH --time=48:00:00
#SBATCH --account=PUT_YOUR_WCID_HERE
#SBATCH --job-name=aria
#SBATCH --partition=batch
nodes=$SLURM_JOB_NUM_NODES
cores=36
module load sierra
export OMPI_MCA_rmaps_base_oversubscribe=1
mpiexec --oversubscribe \
--bind-to core:overload-allowed \
--npernode $cores --n $(($cores*$nodes)) aria --mpmd_master \
--mpi_color 9999 -i input.i : \
--bind-to core:overload-allowed \
--npernode $cores --n $(($cores*$nodes)) other_app [other_app args...]
Contact sierra-help@sandia.gov if you need more help or encounter issues running MPMD jobs.