6.1. Running Fuego

6.1.1. Loading the Sierra Module

Once you have finished setting up an input file, you are ready to run Fuego. From a CEE or HPC UNIX environment at Sandia, you can load the sierra module to access the latest release of Fuego.

$ module load sierra

This will load the current release of Sierra. To load other versions of Sierra, you can use the following modules

  • module load sierra/x.x - Load version x.x (e.g. module load sierra/5.10)

  • module load sierra/sprint - Load the latest sprint release (every 3-week release)

  • module load sierra/daily - Load the daily build of Sierra.

Warning

Using the sierra/daily module exposes you to potential bugs and instabilities since it is actively developed. If the nightly Sierra build process fails, the sierra/daily executable may not exist, or may be much older than expected.

To see a list of the available Sierra versions (and other useful modules available like apps/anaconda3 or apps/matlab) you can use

$ module avail

6.1.2. Running Fuego Locally

To run a job on a non-queued system (e.g. a CEE blade or a cee compute machine) you can call launch or mpirun. For example, to run a job on 4 processors, you would use

$ module load sierra
$ launch -n 4 fuego -i demo.i

The launch command is usually equivalent to using mpirun for local execution, but handles setting required MPI flags when running on some HPC systems.

$ module load sierra
$ mpirun -np 4 fuego -i demo.i

6.1.3. Using the Sierra Script

The sierra script included in the sierra modules provides some additional functionality for launching sierra jobs. Its use is similar to the launch and mpirun commands.

$ sierra --np 4 fuego -i demo.i

By default, the sierra script will perform extra steps that are not necessary for running Fuego. These include:

  • Reading the input file to find the mesh file and running decomp on it beforehand.

  • Reading the input file to find the output files and running epu on them after the simulation is done.

Fuego will automatically decompose your mesh, which is usually faster than manually running decomp, and most visualization tools can view decomposed output files so running epu to combine them into a single file is usually not needed either. To use the sierra script without invoking those steps, add the --run option.

$ sierra --run --np 4 fuego -i demo.i

6.1.4. Running Fuego on an HPC

The HPC systems at Sandia use slurm to schedule jobs. To run a job on an HPC you need to submit it to the slurm system along with some additional information (listed below) and the system will put your job in the queue and run at some point later in the future.

  • The number of compute nodes to run on.

  • The number of cores to use per node.

  • The wall-clock duration of the job (slurm will kill the job if it’s not done by this time limit).

  • A WCID for the job based on the project funding it. The WCID is used for tracking purposes and also determines the job priority. Use the WC Tool web site to check your WCIDs or get a new one.

  • Which queue to submit to (most HPCs have “batch”, “short”, and “long” - “batch” is the standard).

Refer to the HPC homepage for details about queue limits, core counts per node, and other useful HPC information for the machine you intend to run on.

If you are using SAW to submit your jobs, it can handle collecting the required information and submitting the job to the queue. If you do not use SAW, you can submit your jobs manually using either a batch script or the sierra script.

For simple job submissions you can use the sierra script directly after logging in to the HPC you want to run on. For example, to run Fuego on 360 processors for up to 24 hours you would log in to the HPC and run the following command.

$ sierra --run --np 360 fuego -i demo.i --account WCID --queue-name batch --time-limit 24:00:00

For more complicated submissions, you may need to prepare a batch script to perform any custom pre-processing steps you need. There are example submission scripts for different platforms in /projects/samples at Sandia. An example script to run Fuego is shown below.

#!/bin/bash
#SBATCH --nodes=10                    # Number of nodes
#SBATCH --ntasks-per-node=36          # Number of cores per node
#SBATCH --time=24:00:00               # Wall clock time (HH:MM:SS)
#SBATCH --account=PUT_YOUR_WCID_HERE  # WC ID
#SBATCH --job-name=test               # Name of job
#SBATCH --partition=batch             # partition/queue name: short or batch

nodes=$SLURM_JOB_NUM_NODES
cores=36

# do any pre-processing steps you need

mpiexec --bind-to core --npernode $cores --n $(($cores*$nodes)) fuego -i demo.i

If you saved the above script as run_fuego then you would submit it to the queue using

$ sbatch run_fuego

You can check on the status of your queued jobs using squeue -u myusername. To see an estimate of when they will start, use squeue -u myusername --start.