Motion primitives allow for application of discrete search algorithms to rapidly produce trajectories in complex continuous space. The maneuver automaton (MA) provides an elegant formulation for creating a primitive library based on trims and maneuvers. However, performance is fundamentally limited by the contents of the primitive library. If the library is too sparse, performance can be poor in terms of path cost, whereas a library that is too large can increase run time. This work outlines new methods for using genetic algorithms to prune a primitive library. The proposed methods balance the path cost and planning time while maintaining the reachability of the MA. The genetic algorithm in this paper evaluates and mutates populations of motion primitive libraries to optimize both objectives. Here, we illustrate the performance of these methods with a simulated study using a nonlinear medium-fidelity F-16 model. We optimize a library with the presented algorithm for obstacle-free navigation and a nap-of-the-Earth navigation task. In the obstacle-free navigation task, we show a tradeoff of a 10.16% higher planning cost for a 96.63% improvement in run time. In the nap-of-the-Earth task, we show a tradeoff of a 9.712% higher planning cost for a 92.06% improvement in run time.
Motion primitives provide an approach to kinodynamic path planning that does not require online solution of the equations of motion, permitting the use of complex high-fidelity models. The path planning problem with motion primitives is a Markov Decision Process (MDP) with the primitives defining the available actions. Uncertainty in evolution of the primitives means that there is uncertainty in the state resulting from each action. In this work, uncertain motion primitive planning is demonstrated for a high speed glide vehicle. A nonlinear 6- degree-of-freedom model is used to generate the primitive library, and the motion primitive planning problem is formulated so that the transition probabilities in the MDP may have a functional dependence on the state of the system. Single-query solutions to planning problems incorporating operational envelope constraints and no-fly zones are obtained using AO* under chance constraints on the risk of mission failure.
Motion primitives (MPs) provide a fundamental abstraction of movement templates that can be used to guide and navigate a complex environment while simplifying the movement actions. These MPs, when utilized as an action space in reinforcement learning (RL), can allow an agent to learn to select a sequence of simple actions to guide a vehicle towards desired complex mission outcomes. This is particularly useful for missions involving high speed aerospace vehicles (HSAVs) (i.e., Mach 1 to 30) where near real time trajectory generation is needed but the computational cost and timeliness of trajectory generation remains prohibitive. This paperdemonstrates that when MPs are employed in conjunction with RL, the agent can learn to solve a wider range of problems for HSAV missions. To this end, using both a MP and and non-MP approach, RL is employed to solve the problem of an HSAV arriving at a non-maneuvering moving target at a constant altitude and with an arbitrary, but constant, velocity and heading angle. The MPs for HSAV consist of multiple pull (flight path angle) and turn (heading angle) commands that are defined for a specific duration based on mission phases; whereas the non-MP approach uses angle of attack and bank angle as action space for RL. The paper describes details on HSAV problem formulation to include equations of motion, observation space, telescopic reward function, RL algorithm and hyperparameters, RL curriculum, formation of the MPs, and calculation of time to execute the MP used for the problem. Our results demonstrate that the non-MP approach is unable to even train an agent that is successful in the base-case of the RL curriculum. The MP approach, however, can train an agent with success rate of 76.6% inarriving at a target moving with any heading angle with a velocity between 0 and 500 m/s.
Raz, Ali K.; Nolan, Sean M.; Levin, Winston; Mall, Kshitij; Mia, Ahmad; Mockus, Linas; Ezra, Kris; Williams, Kyle R.
Reinforcement Learning (RL) provides an ability to train an artificial intelligent agent in dynamic and uncertain environments. RL has demonstrated an impressive performance capability to learn nearly optimal policies in various application domains including aerospace. Despite the demonstrated performance outcomes of RL, characterizing performance boundaries, explaining the logic behind RL decisions, and quantifying resulting uncertainties in RL outputs are major challenges that slow down the adoption of RL in real-time systems. This is particularly true for aerospace systems where the risk of failure is high and performance envelopes of systems of interest may be small. To facilitate adoption of learning agents in real-time systems, this paper presents a three-part Test and Evaluation (T&E) framework for RL built from Systems engineering for artificial intelligence (SE4AI) perspective. This T&E framework introduces robustness testing approaches to characterize performance bounds on RL, employs Explainable AI techniques, namely Shapley Additive Explanations (SHAP) to examine RL decision-making, and incorporates validation of RL outputs with known and accepted solutions. This framework is applied to a high-speed aerospace vehicle emergency descent problem where RL is trained to provide an angle of attack command and the framework is utilized to comprehensively examine the impact of uncertainties in the vehicle's altitude, velocity, and flight path angle. The robustness testing characterizes acceptable ranges of disturbances in flight parameters, while SHAP exposes the most significant features that impact RL selection of angle of attack-in this case the vehicle altitude. Finally, RL outputs are compared to trajectory generated by indirect optimal control methods for validation.
This paper describes how the performance of motion primitive-based planning algorithms can be improved using reinforcement learning. Specifically, we describe and evaluate a framework that autonomously improves the performance of a primitive-based motion planner. The improvement process consists of three phases: exploration, extraction, and reward updates. This process can be iterated continuously to provide successive improvement. The exploration step generates new trajectories, and the extraction step identifies new primitives from these trajectories. These primitives are then used to update rewards for continued exploration. This framework required novel shaping rewards, development of a primitive extraction algorithm, and modification of the Hybrid A* algorithm. The framework is tested on a navigation task using a nonlinear F-16 model. The framework autonomously added 91 motion primitives to the primitive library and reduced average path cost by 21.6 s, or 35.75% of the original cost. The learned primitives are applied to an obstacle field navigation task, which was not used in training, and reduced path cost by 16.3 s, or 24.1%. Additionally, two heuristics for the modified Hybrid A* algorithm are designed to improve effective branching factor.
Multi-phase, pseudospectral optimization is employed in a variety of applications, but many of the world-class optimization libraries are closed-source. In this paper we formulate an open-source, object-oriented framework for dynamic optimization using the Pyomo modeling language. This strategy supports the reuse of common code for rapid, error-free model development. Flexibility of our framework is demonstrated on a series of dynamic optimization problems, including multi-phase trajectory optimization using highly accurate pseudospectral methods and controller gain optimization in the presence of stability margin constraints. We employ numerical procedures to improve convergence rates and solution accuracy. We validate our framework using GPOPS-II, a commercial, MATLAB-based optimization program, for a vehicle ascent problem. The trajectory results show close alignment with this state-of-the-art optimization suite.
Pyomo and Dakota are openly available software packages developed by Sandia National Labs. In this tutorial, methods for automating the optimization of controller parameters for a nonlinear cart-pole system are presented. Two approaches are described and demonstrated on the cart-pole example problem for tuning a linear quadratic regulator and also a partial feedback linearization controller. First the problem is formulated as a pseudospectral optimization problem under an open box methodology utilizing Pyomo, where the plant model is fully known to the optimizer. In the next approach, a black-box approach utilizing Dakota in concert with a MATLAB or Simulink plant model is discussed, where the plant model is unknown to the optimizer. A comparison of the two approaches provides the end user the advantages and shortcomings of each method in order to pick the right tool for their problem. We find that complex system models and objectives are easily incorporated in the Dakota-based approach with minimal setup time, while the Pyomo-based approach provides rapid solutions once the system model has been developed.