4.14.2. Plugin Utilities

Warning

The use of user utilities should be considered a last resort, especially for production simulations - most custom capabilities can be implemented using GPU-compatible, well-tested capabilities like string functions.

User utilities are another way that users can extend Fuego’s capabilities using custom compiled code. Unlike a user subroutine, which is evaluated with a very controlled and limited API, a user utility is a standalone class with an execute function that can be executed at various points during problem execution. A user subroutine cannot generally access field data or perform MPI calls, but a user utility can do both of those things.

The tradeoff is that the API for a user utility is much more general and open, and as such cannot be completely documented here. This section will provide some guidelines on how to define them, and some example utilities. For more advanced capabilities, you will likely need to involve a Fuego developer.

4.14.2.1. Writing a Utility

In this section we’ll write an example utility to calculate a custom field we will use for a boundary flux. The first part of our plugin C file will define the utility class. The API requirements for this class are:

  1. It must inherit from sierra::Afgo::UserUtility

  2. Its constructor must use the following signature - MyUtil(sierra::Afgo::Region& r)

  3. It must define a static String name() method

  4. It must define an execute function as void execute() override

  5. It may define a field registration function as void register_utility_variables(const stk::mesh::ConstPartVector& parts) override, but this is not required

Our class definition looks like:

#include "Afgo_UserPlugins.h"
#include "Afgo_DiagWriter.h"
#include "tftk_mesh/tftk_Mesh.h"
#include "sierra/Sierra_FieldType.h"
#include "tftk_mesh/tftk_FieldFunctions.h"

using namespace sierra::Afgo;

class MyCustomUtil : public UserUtility
{
public:
  static sierra::String name() { return "MyCustomUtil"; }
  MyCustomUtil(Region& r);
  void execute() override;
  void register_utility_variables(const stk::mesh::ConstPartVector& parts) override;
};

The next part of the utility C file is the constructor definition. This is where you define when in the execution sequence it should be run. Common choices for when the utility should be run are:

  • INITIAL_WORK - Run only once during initialization (before the first time step)

  • PRE_TSTEP - Run at the start of each time step before any Newton iteration

  • PRE_ITER - Run before beginning nonlinear iteration at each time step

  • PRE_SOLVE - Run before each nonlinear iteration

  • POST_TSTEP - Run at the end of each time step

  • POST_ITER - Run after finishing nonlinear iteration at each time step

  • POST_SOLVE - Run after each nonlinear iteration

MyCustomUtil::MyCustomUtil(Region& r)
: UserUtility( name(), r, UserUtility::UtilExecution::PRE_SOLVE )
{}

The next components of the plugin C file are the definitions for the other functions in the utility

void MyCustomUtil::register_utility_variables(const stk::mesh::ConstPartVector& parts) override {
  afgoout << "Registering my custom fields\n";
  mesh().register_field("some_field", sierra::FieldType::REAL,
    stk::topology::NODE_RANK, sierra::TEMPORARY, 1, parts
  );
}

void MyCustomUtil::execute()
{
  afgoout << "Running my custom utility\n";

  auto my_output = mesh().get_node_field("some_field");
  auto temp = mesh().get_node_field("temperature");

  auto selector = mesh().active_not_aura_selector() & stk::mesh::selectField(my_flux);

  using F = stk::mesh::NgpField<double>;
  tftk::mesh::for_each_entity_run(
    "DoSomething",
    mesh(),
    selector,
    my_output,
    {temp},
    KOKKOS_LAMBDA(stk::mesh::FastMeshIndex mi, const F& out, const F& T) {
      out(mi,0) = 10 + 4*T(mi,0);
    },
    tftk::HostSpace()
  );
}

Finally, we need to register the utility. This is done with the following command:

extern "C" void register_usersubs() {
  sierra::Afgo::UtilityPluginFactory<MyCustomUtil> register_my_util{};
}

The name of the variable here (register_my_util) does not matter.

4.14.2.2. Compiling a Utility

User utilities are compiled to shared object (.so) files using the same procedure as User Subroutines. The following command will scan the input file for import .so files and compile them from source files of the same name.

sierra --make fuego -i input.i

4.14.2.3. Using a Utility

In the Fuego input file, the plugin is loaded using the command (assuming the plugin is defined in My_Utility_Plugin.C)

Begin Sierra My_Sierra_Job
   Load User Plugin file ./My_Utility_Plugin.so USING FUNCTION register_usersubs

No changes are needed in the input file to use a plugin utility, it is automatically added to the requested spot and executed.