# 6.1. Running Fuego

## 6.1.1. Loading the Sierra Module

Once you have finished setting up an input file, you are ready to run Fuego. From a CEE or HPC UNIX environment at Sandia, you can load the sierra module to access the latest release of Fuego.

    $ module load sierra

This will load the current release of Sierra. To load other versions of Sierra, you can use the following modules

> - `module load sierra/x.x` - Load version `x.x` (e.g. `module load sierra/5.10`)

> - `module load sierra/sprint` - Load the latest sprint release (every 3-week release)

> - `module load sierra/daily` - Load the daily build of Sierra.

Warning

Using the `sierra/daily` module exposes you to potential bugs and instabilities since it is actively developed. If the nightly Sierra build process fails, the `sierra/daily` executable may not exist, or may be much older than expected.

To see a list of the available Sierra versions (and other useful modules available like `apps/anaconda3` or `apps/matlab`) you can use

    $ module avail

## 6.1.2. Running Fuego Locally

To run a job on a non-queued system (e.g. a CEE blade or a cee compute machine) you can call `launch` or `mpirun`. For example, to run a job on 4 processors, you would use

    $ module load sierra
    $ launch -n 4 fuego -i demo.i

The `launch` command is usually equivalent to using `mpirun` for local execution, but handles setting required MPI flags when running on some HPC systems.

    $ module load sierra
    $ mpirun -np 4 fuego -i demo.i

## 6.1.3. Using the Sierra Script

The `sierra` script included in the sierra modules provides some additional functionality for launching sierra jobs. Its use is similar to the `launch` and `mpirun` commands.

    $ sierra --np 4 fuego -i demo.i

By default, the sierra script will perform extra steps that are not necessary for running Fuego. These include:

> - Reading the input file to find the mesh file and running `decomp` on it beforehand.

> - Reading the input file to find the output files and running `epu` on them after the simulation is done.

Fuego will automatically decompose your mesh, which is usually faster than manually running `decomp`, and most visualization tools can view decomposed output files so running `epu` to combine them into a single file is usually not needed either. To use the sierra script without invoking those steps, add the `--run` option.

    $ sierra --run --np 4 fuego -i demo.i

## 6.1.4. Running Fuego on an HPC

The HPC systems at Sandia use slurm to schedule jobs. To run a job on an HPC you need to submit it to the slurm system along with some additional information (listed below) and the system will put your job in the queue and run at some point later in the future.

> - The number of compute nodes to run on.

> - The number of cores to use per node.

> - The wall-clock duration of the job (slurm will kill the job if it’s not done by this time limit).

> - A WCID for the job based on the project funding it. The WCID is used for tracking purposes and also determines the job priority. Use the WC Tool web site to check your WCIDs or get a new one.

> - Which queue to submit to (most HPCs have “batch”, “short”, and “long” - “batch” is the standard).

Refer to the HPC homepage for details about queue limits, core counts per node, and other useful HPC information for the machine you intend to run on.

If you are using SAW to submit your jobs, it can handle collecting the required information and submitting the job to the queue. If you do not use SAW, you can submit your jobs manually using either a batch script or the sierra script.

For simple job submissions you can use the sierra script directly after logging in to the HPC you want to run on. For example, to run Fuego on 360 processors for up to 24 hours you would log in to the HPC and run the following command.

    $ sierra --run --np 360 fuego -i demo.i --account WCID --queue-name batch --time-limit 24:00:00

For more complicated submissions, you may need to prepare a batch script to perform any custom pre-processing steps you need. There are example submission scripts for different platforms in `/projects/samples` at Sandia. An example script to run Fuego is shown below.

    #!/bin/bash
    #SBATCH --nodes=10                    # Number of nodes
    #SBATCH --ntasks-per-node=36          # Number of cores per node
    #SBATCH --time=24:00:00               # Wall clock time (HH:MM:SS)
    #SBATCH --account=PUT_YOUR_WCID_HERE  # WC ID
    #SBATCH --job-name=test               # Name of job
    #SBATCH --partition=batch             # partition/queue name: short or batch

    nodes=$SLURM_JOB_NUM_NODES
    cores=36

    # do any pre-processing steps you need

    mpiexec --bind-to core --npernode $cores --n $(($cores*$nodes)) fuego -i demo.i

If you saved the above script as `run_fuego` then you would submit it to the queue using

    $ sbatch run_fuego

You can check on the status of your queued jobs using `squeue -u myusername`. To see an estimate of when they will start, use `squeue -u myusername --start`.

## 6.1.5. Running Fuego on Hops

The Hops (SRN) cluster is comprised of 64 nodes, each with 4x 80 GB H100 NVIDIA GPUs. Compared to traditional HPC clusters like Attaway / Eclipse / Amber / etc., Hops also uses the slurm scheduler, however, each node has significantly more memory bandwidth / compute available. The following are general recommendations for running on Hops, along with example submission batch scripts for both regular MPI jobs and MPMD jobs.

Recommendations:

> - Nodes are powerful but limited in number, try to fully utilize them. As a starting point, target around 15M-20M DOFs per linear system per compute node. For Fuego, DOFs are always at the mesh nodes so this is equivalent to sizing 15M-20M mesh nodes per compute node.

> Note

> Excluding the cost of the first step, node for node you should see order of magnitude speedups 10-20x relative to Eclipse (CTS-1) and 2.5-5x speedups relative to Amber (CTS-2). If you do not see such speedups read below.

> Warning

> Fuego is designed to run with 1 MPI rank per GPU, or in MPMD mode with PMR, at most 2 MPI ranks per GPU. In regular MPI submissions, do not run more than 1 task per GPU, as performance will degrade due to context switching. Use `nvidia-smi` to verify tasks are split properly across GPUs.

> - Configure models on host platforms (e.g. cee / amber, etc) as some of the initialization occurs on host when using Hops. Since we are running with only four MPI ranks per node, this initialization cost can be significant compared to host platforms that run on many more ranks per node (e.g., Amber has 112 ranks per node).

> - Fuego is partially converted, meaning some algorithms may run on host. To check if there are algorithms running on host, pass `--afgoout debug` to Fuego to write out a list of host algorithms to the log file, then in a terminal `cat fuego.log | grep "Creating host"`. Depending on the algorithms listed, they may severely degrade overall performance.

> Note

> As example, consider a volume host algorithm that is running on 1 node of Hops vs 1 node of Amber. Excluding the added cost of syncing fields (potentially also the LHS / RHS) to host and back to device, a rough estimate of the respective slowdown is 112 / 4 since we run with 1 MPI rank per GPU on Hops but 112 ranks per node on Amber. If on the other hand, it is a surface algorithm running on a small portion of your domain (and does not modify LHS), you may potentially get away with running the host algorithm on Hops. A question one might have is, can we thread the legacy host algorithms to fully utilize the remaining CPU cores, the answer is threading of host algorithms requires the same Kokkos conversion that would be required to run on GPU, in which case it would be more beneficial to simply run on the GPU after conversion of the host algorithm.

> Warning

> All particle physics runs on the host at the moment. We do not have a timeline on when this feature will be converted to run on GPUs.

> - Check your preconditioners. If you specify to use the SGS preconditioner, we will respect that irrespective of the potential performance penalties. SGS is not GPU friendly. For scalar transport problems that are diagonally dominant we recommend to start with 1-3 sweeps of the Jacobi preconditioner. If you observe linear convergence issues, switch to SGS2 preconditioner which is also GPU friendly.

> - If you suspect host algorithms are hot spots in your GPU runs, email us and we will determine a priority to convert the algorithm. If no host algorithms are reported and you do not see order of magnitude speedups relative to Eclipse, email us at sierra-help@sandia.gov so we can diagnose your issue.

To submit a regular MPI job fully utilizing the four gpus on each node you can call the following script using sbatch

    #!/bin/bash
    #SBATCH --account=fyXXXXX  # WC ID
    #SBATCH --job-name=test
    #SBATCH --partition=batch
    #SBATCH --qos=normal

    # at the moment only daily is installed and we are waiting on point release to be installed
    module load sierra/daily

    num_nodes=$SLURM_JOB_NUM_NODES
    task_per_node=4
    ntasks=$((task_per_node * num_nodes))

    launch -n $ntasks fuego -i input.i

    # alternatively
    # srun -n $ntasks fuego -i input.i

The following sbatch command will submit to two nodes, running 4 tasks split across the available gpus, for a total of 8 tasks (MPI ranks) reserving 60 minutes.

    $ sbatch -N 2 -t 60 submit_hops.sh

As mentioned earlier, for regular MPI jobs, each task should bind exactly to one gpu, otherwise if multiple tasks are bound per GPU a performance penalty may be incurred due to context switching. To verify, you have the correct distribution of tasks, ssh to a compute node where your job is running and use `nvidia-smi`

    $ squeue -u [username]
    $ ssh hops[node]
    $ nvidia-smi

For this example, for a given node, the `nvidia-smi` output should show 4 tasks running, each on their own gpu along with the memory consumed per GPU.

Similarly, to submit a MPMD job you can also use the launch wrapper

    #!/bin/bash
    #SBATCH --account=fyXXXXX  # WC ID
    #SBATCH --job-name=test
    #SBATCH --partition=batch
    #SBATCH --qos=normal

    module load sierra/daily

    num_nodes=$SLURM_JOB_NUM_NODES
    fuego_task_per_node=4
    pmr_task_per_node=4

    ntasks_fuego=$((fuego_task_per_node * num_nodes))
    ntasks_pmr=$((pmr_task_per_node * num_nodes))

    launch -n $ntasks_fuego fuego -i fuego_input.i : -n $ntasks_pmr pmr -i pmr_input.i

where the following sbatch command will split tasks in a round robin fashion across available GPUs.

    $ sbatch -N 2 --distribution=cyclic -t 60 submit_mpmd_hops.sh

Using `nvidia-smi`, you will see each node has 4 tasks of Fuego and 4 tasks of PMR, with one task of Fuego and one task of PMR per GPU for a total of 16 tasks across the 8 GPUs. Fuego and PMR alternate in work and so it is fine to overload each GPU in this case, though each GPU will consume more memory.

Note

The launch wrapper automatically outputs a temporary MPMD config file and passes that file to `srun` i.e.,

    srun -n $total_tasks --multi-prog mpmd_config_file

where the MPMD config file specifies tasks per app i.e.,

    0-7 fuego -i fuego_input.i
    8-15 pmr -i pmr_input.i

With this MPMD config file, without passing `--distribution=cyclic` to `srun`, the first 8 tasks of Fuego will land on the first node and the second 8 tasks of PMR will land on the second node. This is not ideal as each GPU will need to frequently context switch between different Fuego (node 1) or PMR (node 2) tasks.

Note

PMR is fully converted to the GPU with the exception of minor initialization costs.

## 6.1.6. Running Fuego on Eldorado

The Eldorado (SRN) cluster is comprised of 382 nodes, each with 4x 128GB MI300A AMD GPUs. Many of our recommendations for running on Hops directly carry over to running on Eldorado. However, there are major differences that we expect to impact user submissions, these include:

- job submissions are handled via the flux scheduler

- each MI300A can be logically partitioned into 6 GPUs (CPX mode)

By default, Sierra assumes CPX mode is being used and exports MPI bindings such that either one task (SPMD) or two tasks (MPMD) binds to one XCD. In CPX mode, each MI300A has 6 XCDs and so to fully utilize an Eldorado node, 24 tasks (SPMD) or 48 tasks (MPMD) should be used per node.

The CPX hardware partitioning results in increased parallelism and decreases compute costs for any host paths remaining in the code. As such, compared to Hops, users should see quicker initialization and improved performance when running partially converted input decks (i.e., unconverted host algorithms).

With respect to performance, current performance is similar to Hops and users should continue to follow the GPU submission guidelines i.e., fit as large of a problem per node as possible. Compared to Hops, larger problems can be fit per node due to the increased memory of the MI300A (512 GB total per node). Over time we expect performance should improve, but for now the previous guidelines of 10-20x speedups relative to Eclipse (CTS-1) and 2.5-5x speedups relative to Amber (CTS-2) can be used as an expected performance baseline. If you do not see such speedups, email sierra-help@sandia.gov. Note that these speedups are node for node.

To submit a SPMD job on Eldorado via the flux scheduler the following script can be used

    #!/bin/bash

    #flux: --job-name=my_spmd_job
    #flux: --output='my_spmd_job.{{id}}.out'
    #flux: --error='my_spmd_job.{{id}}.err'
    #flux: --setattr=gpumode=CPX # enable CPX mode
    #flux: --conf=resource.rediscover=true
    #flux: -x # exclusive access to resources
    #flux: -q pbatch

    module load sierra/5.26

    num_nodes=$(flux resource info | grep -oP '\d+ Nodes' | awk '{print $1}')
    fuego_task_per_node=24 # SPMD jobs should use 24 tasks per node to fully utilize CPX mode
    ntasks_fuego=$((fuego_task_per_node * num_nodes))

    launch -n $ntasks_fuego fuego -i input.i

where the following flux batch command can be used to submit 24 tasks of Fuego on 1 node for 60 minutes.

    $ flux batch -N 1 -t 60m submit_spmd_eldorado.sh

To see a list of running jobs, use

    $ flux jobs

Similarly, to submit a MPMD job on Eldorado, the following script can be used

    #!/bin/bash
    #flux: --job-name=my_mpmd_job
    #flux: --output='my_mpmd_job.{{id}}.out'
    #flux: --error='my_mpmd_job.{{id}}.err'
    #flux: --setattr=gpumode=CPX # enable CPX mode
    #flux: --conf=resource.rediscover=true
    #flux: -x # exclusive access to resources
    #flux: -q pbatch

    module load sierra/5.26

    num_nodes=$(flux resource info | grep -oP '\d+ Nodes' | awk '{print $1}')
    fuego_task_per_node=24
    pmr_task_per_node=24

    ntasks_fuego=$((fuego_task_per_node * num_nodes))
    ntasks_pmr=$((pmr_task_per_node * num_nodes))

    launch -n $ntasks_fuego fuego -i fuego_input.i : -n $ntasks_pmr pmr -i pmr_input.i

where the flux batch command to submit the 192 task job (96 tasks for Fuego and 96 tasks for PMR) using 96 GPUs (XCDs) spread across 4 nodes for 2 hours is

    $ flux batch -N 4 -t 2h submit_mpmd_eldorado.sh

Note

- The `launch` command automatically maps SPMD or MPMD commands to the appropriate `flux run ...` commands, ensuring tasks bind properly to the appropriate GPUs. The `flux run` command used is shown in the flux output and should be using `--core-per-task=4` for SPMD and `--cores-per-task=2` for MPMD, which will result in either 1 task or 2 tasks binding per XCD, respectively.

- For MPMD, the launch wrapper will also automatically output a temporary MPMD config file that is used to assign one task of each app per XCD in a round robin fashion.

Warning

- CPX mode MUST be enabled. Trying to `flux run` directly on a login node or outside of a CPX allocation will error due to the fact modules currently automatically export an MPI bind topology file that assumes CPX mode is enabled.