Multi-objective optimization methods can be criticized for lacking a statistically valid measure of the quality and representativeness of a solution. This stance is especially relevant to metaheuristic optimization approaches but can also apply to other methods that typically might only report a small representative subset of a Pareto frontier. Here we present a method to address this deficiency based on random sampling of a solution space to determine, with a specified level of confidence, the fraction of the solution space that is surpassed by an optimization. The Superiority of Multi-Objective Optimization to Random Sampling, or SMORS method, can evaluate quality and representativeness using dominance or other measures, e.g., a spacing measure for high-dimensional spaces. SMORS has been tested in a combinatorial optimization context using a genetic algorithm but could be useful for other optimization methods.
Schedule Management Optimization (SMO) is a tool for automatically generating a schedule of project tasks. Project scheduling is traditionally achieved with the use of commercial project management software or case-specific optimization formulations. Commercial software packages are useful tools for managing and visualizing copious amounts of project task data. However, their ability to automatically generate optimized schedules is limited. Furthermore, there are many real-world constraints and decision variables that commercial packages ignore. Case-specific optimization formulations effectively identify schedules that optimize one or more objectives for a specific problem, but they are unable to handle a diverse selection of scheduling problems. SMO enables practitioners to generate optimal project schedules automatically while considering a broad range of real-world problem characteristics. SMO has been designed to handle some of the most difficult scheduling problems – those with resource constraints, multiple objectives, multiple inventories, and diverse ways of performing tasks. This report contains descriptions of the SMO modeling concepts and explains how they map to real-world scheduling considerations.
As system of systems (SoS) models become increasingly complex and interconnected a new approach is needed to capture the effects of humans within the SoS. Many real-life events have shown the detrimental outcomes of failing to account for humans in the loop. This research introduces a novel and cross-disciplinary methodology for modeling humans interacting with technologies to perform tasks within an SoS specifically within a layered physical security system use case. Metrics and formulations developed for this new way of looking at SoS termed sociotechnical SoS allow for the quantification of the interplay of effectiveness and efficiency seen in detection theory to measure the ability of a physical security system to detect and respond to threats. This methodology has been applied to a notional representation of a small military Forward Operating Base (FOB) as a proof-of-concept.
As system of systems (SoS) models become increasingly complex and interconnected a new approach is needed to capture the effects of humans within the SoS. Many real-life events have shown the detrimental outcomes of failing to account for humans in the loop. This research introduces a novel and cross-disciplinary methodology for modeling humans interacting with technologies to perform tasks within an SoS specifically within a layered physical security system use case. Metrics and formulations developed for this new way of looking at SoS termed sociotechnical SoS allow for the quantification of the interplay of effectiveness and efficiency seen in detection theory to measure the ability of a physical security system to detect and respond to threats. This methodology has been applied to a notional representation of a small military Forward Operating Base (FOB) as a proof-of-concept.
Modern systems, such as physical security systems, are often designed to involve complex interactions of technological and human elements. Evaluation of the performance of these systems often overlooks the human element. A method is proposed here to expand the concept of sensitivity—as denoted by d’—from signal detection theory (Green & Swets 1966; Macmillan & Creelman 2005), which came out of the field of psychophysics, to cover not only human threat detection but also other human functions plus the performance of technical systems in a physical security system, thereby including humans in the overall evaluation of system performance. New in this method is the idea that probabilities of hits (accurate identification of threats) and false alarms (saying “threat” when there is not one), which are used to calculate d’ of the system, can be applied to technologies and, furthermore, to different functions in the system beyond simple yes-no threat detection. At the most succinct level, the method returns a single number that represents the effectiveness of a physical security system; specifically, the balance between the handling of actual threats and the distraction of false alarms. The method can be automated, and the constituent parts revealed, such that given an interaction graph that indicates the functional associations of system elements and the individual probabilities of hits and false alarms for those elements, it will return the d’ of the entire system as well as d’ values for individual parts. The method can also return a measure of the response bias* of the system. One finding of this work is that the d’ for a physical security system can be relatively poor in spite of having excellent d’s for each of its individual functional elements.
Upcoming weapon programs require an aggressive increase in Application Specific Integrated Circuit (ASIC) production at Sandia National Laboratories (SNL). SNL has developed unique modeling and optimization tools that have been instrumental in improving ASIC production productivity and efficiency, identifying optimal operational and tactical execution plans under resource constraints, and providing confidence in successful mission execution. With ten products and unprecedented levels of demand, a single set of shared resources, highly variable processes, and the need for external supplier task synchronization, scheduling is an integral part of successful manufacturing. The scheduler uses an iterative multi-objective genetic algorithm and a multi-dimensional performance evaluator. Schedule feasibility is assessed using a discrete event simulation (DES) that incorporates operational uncertainty, variability, and resource availability. The tools provide rapid scenario assessments and responses to variances in the operational environment, and have been used to inform major equipment investments and workforce planning decisions in multiple SNL facilities.
System-of-systems modeling has traditionally focused on physical systems rather than humans, but recent events have proved the necessity of considering the human in the loop. As technology becomes more complex and layered security continues to increase in importance, capturing humans and their interactions with technologies within the system-of-systems will be increasingly necessary. After an extensive job-task analysis, a novel type of system-ofsystems simulation model has been created to capture the human-technology interactions on an extra-small forward operating base to better understand performance, key security drivers, and the robustness of the base. In addition to the model, an innovative framework for using detection theory to calculate d’ for individual elements of the layered security system, and for the entire security system as a whole, is under development.
Our society is increasingly reliant on systems and interoperating collections of systems, known as systems of systems (SoS). These SoS are often subject to changing missions (e.g., nation- building, arms-control treaties), threats (e.g., asymmetric warfare, terrorism), natural environments (e.g., climate, weather, natural disasters) and budgets. How well can SoS adapt to these types of dynamic conditions? This report details the results of a three year Laboratory Directed Research and Development (LDRD) project aimed at developing metrics and methodologies for quantifying the adaptability of systems and SoS. Work products include: derivation of a set of adaptability metrics, a method for combining the metrics into a system of systems adaptability index (SoSAI) used to compare adaptability of SoS designs, development of a prototype dynamic SoS (proto-dSoS) simulation environment which provides the ability to investigate the validity of the adaptability metric set, and two test cases that evaluate the usefulness of a subset of the adaptability metrics and SoSAI for distinguishing good from poor adaptability in a SoS. Intellectual property results include three patents pending: A Method For Quantifying Relative System Adaptability, Method for Evaluating System Performance, and A Method for Determining Systems Re-Tasking.