The ground truth program used simulations as test beds for social science research methods. The simulations had known ground truth and were capable of producing large amounts of data. This allowed research teams to run experiments and ask questions of these simulations similar to social scientists studying real-world systems, and enabled robust evaluation of their causal inference, prediction, and prescription capabilities. We tested three hypotheses about research effectiveness using data from the ground truth program, specifically looking at the influence of complexity, causal understanding, and data collection on performance. We found some evidence that system complexity and causal understanding influenced research performance, but no evidence that data availability contributed. The ground truth program may be the first robust coupling of simulation test beds with an experimental framework capable of teasing out factors that determine the success of social science research.
Measures of simulation model complexity generally focus on outputs; we propose measuring the complexity of a model’s causal structure to gain insight into its fundamental character. This article introduces tools for measuring causal complexity. First, we introduce a method for developing a model’s causal structure diagram, which characterises the causal interactions present in the code. Causal structure diagrams facilitate comparison of simulation models, including those from different paradigms. Next, we develop metrics for evaluating a model’s causal complexity using its causal structure diagram. We discuss cyclomatic complexity as a measure of the intricacy of causal structure and introduce two new metrics that incorporate the concept of feedback, a fundamental component of causal structure. The first new metric introduced here is feedback density, a measure of the cycle-based interconnectedness of causal structure. The second metric combines cyclomatic complexity and feedback density into a comprehensive causal complexity measure. Finally, we demonstrate these complexity metrics on simulation models from multiple paradigms and discuss potential uses and interpretations. These tools enable direct comparison of models across paradigms and provide a mechanism for measuring and discussing complexity based on a model’s fundamental assumptions and design.
The causal structure of a simulation is a major determinant of both its character and behavior, yet most methods we use to compare simulations focus only on simulation outputs. We introduce a method that combines graphical representation with information theoretic metrics to quantitatively compare the causal structures of models. The method applies to agent-based simulations as well as system dynamics models and facilitates comparison within and between types. Comparing models based on their causal structures can illuminate differences in assumptions made by the models, allowing modelers to (1) better situate their models in the context of existing work, including highlighting novelty, (2) explicitly compare conceptual theory and assumptions to simulated theory and assumptions, and (3) investigate potential causal drivers of divergent behavior between models. We demonstrate the method by comparing two epidemiology models at different levels of aggregation.
Social systems are uniquely complex and difficult to study, but understanding them is vital to solving the world’s problems. The Ground Truth program developed a new way of testing the research methods that attempt to understand and leverage the Human Domain and its associated complexities. The program developed simulations of social systems as virtual world test beds. Not only were these simulations able to produce data on future states of the system under various circumstances and scenarios, but their causal ground truth was also explicitly known. Research teams studied these virtual worlds, facilitating deep validation of causal inference, prediction, and prescription methods. The Ground Truth program model provides a way to test and validate research methods to an extent previously impossible, and to study the intricacies and interactions of different components of research.
This project studied the potential for multiscale group dynamics in complex social systems, including emergent recursive interaction. Current social theory on group formation and interaction focuses on a single scale (individuals forming groups) and is largely qualitative in its explanation of mechanisms. We combined theory, modeling, and data analysis to find evidence that these multiscale phenomena exist, and to investigate their potential consequences and develop predictive capabilities. In this report, we discuss the results of data analysis showing that some group dynamics theory holds at multiple scales. We introduce a new theory on communicative vibration that uses social network dynamics to predict group life cycle events. We discuss a model of behavioral responses to the COVID-19 pandemic that incorporates influence and social pressures. Finally, we discuss a set of modeling techniques that can be used to simulate multiscale group phenomena.
It is widely believed that one's peers influence product adoption behaviors. This relationship has been linked to the number of signals a decision-maker receives in a social network. But it is unclear if these same principles hold when the "pattern" by which it receives these signals vary and when peer influence is directed towards choices which are not optimal. To investigate that, we manipulate social signal exposure in an online controlled experiment using a game with human participants. Each participant in the game decides among choices with differing utilities. We observe the following: (1) even in the presence of monetary risks and previously acquired knowledge of the choices, decision-makers tend to deviate from the obvious optimal decision when their peers make a similar decision which we call the influence decision, (2) when the quantity of social signals vary over time, the forwarding probability of the influence decision and therefore being responsive to social influence does not necessarily correlate proportionally to the absolute quantity of signals. To better understand how these rules of peer influence could be used in modeling applications of real world diffusion and in networked environments, we use our behavioral findings to simulate spreading dynamics in real world case studies. We specifically try to see how cumulative influence plays out in the presence of user uncertainty and measure its outcome on rumor diffusion, which we model as an example of sub-optimal choice diffusion. Together, our simulation results indicate that sequential peer effects from the influence decision overcomes individual uncertainty to guide faster rumor diffusion over time. However, when the rate of diffusion is slow in the beginning, user uncertainty can have a substantial role compared to peer influence in deciding the adoption trajectory of a piece of questionable information.
SIGNAL is a first of its kind experimental wargame developed as part of the Project on Nuclear Gaming (PoNG). In this document we describe the rules and game mechanics associated with the online version of SIGNAL created by team members from the University of California, Berkeley, Sandia National Laboratories, and Lawrence Livermore National Laboratory and sponsored by the Carnegie Corporation of New York. The game was developed as part of a larger research project to develop the experimental wargaming methodology and explore its use on a model scenario: the impact of various military capabilities on conflict escalation dynamics. We discuss the results of this research in a forthcoming paper that will include this manual as an appendix. It is our hope that this manual will both contribute to our players' understanding of the game prior to play and that it will allow for replication of the SIGNAL game environment for future research purposes. The manual begins by introducing the terminology used throughout the document. It then outlines the technical requirements required to run SIGNAL. The following section provides a description of the map, resources, infrastructure, tokens, and action cards used in the game environment. The manual then describes the user interface including the chat functions, trade mechanism, currency and population counts necessary for players to plan their actions. It then turns to the sequence of player actions in the game describing the signaling, action, and upkeep phases that comprise each round of play. It then outlines the use of diplomacy including alliances with minor states and trade between players. The manual also describes the process for scoring the game and determining the winner. The manual concludes with tips for players to remember as they embark upon playing the game.
We develop a methodology for comparing two or more agent-based models that are developed for the same domain, but may differ in the particular data sets (e.g., geographical regions) to which they are applied, and in the structure of the model. Our approach is to learn a response surface in the common parameter space of the models and compare the regions corresponding to qualitatively different behaviors in the models. As an example, we develop an active learning algorithm to learn phase transition boundaries in contagion processes in order to compare two agent-based models of rooftop solar panel adoption.
There is a wealth of psychological theory regarding the drive for individuals to congregate and form social groups, positing that people may organize out of fear, social pressure, or even to manage their self-esteem. We evaluate three such theories for multi-scale validity by studying them not only at the individual scale for which they were originally developed, but also for applicability to group interactions and behavior. We implement this multi-scale analysis using a dataset of communications and group membership derived from a long-running online game, matching the intent behind the theories to quantitative measures that describe players’ behavior. Once we establish that the theories hold for the dataset, we increase the scope to test the theories at the higher scale of group interactions. Despite being formulated to describe individual cognition and motivation, we show that some group dynamics theories hold at the higher level of group cognition and can effectively describe the behavior of joint decision making and higher-level interactions.
National security decisions are driven by complex, interconnected contextual, individual, and strategic variables. Modeling and simulation tools are often used to identify relevant patterns, which can then be shaped through policy remedies. In the paper to follow, however, we argue that models of these scenarios may be prone to the complexity-scarcity gap, in which relevant scenarios are too complex to model from first principles and data from historical scenarios are too sparse-making it difficult to draw representative conclusions. The result are models that are either too simple or are unduly biased by the assumptions of the analyst. We outline a new method of quantitative inquiry-experimental wargaming-as a means to bridge the complexity-scarcity gap that offers human-generated, empirical data to inform a variety of model and simulation tasks (model building, calibration, testing, and validation). Below, we briefly describe SIGNAL-our first-of-a-kind experimental wargame designed to study strategic stability in conflict settings with nuclear weapons. We then highlight the potential utility of this data for modeling and simulation efforts in the future using this data.
It is widely believed that the adoption behavior of a decision-maker in a social network is related to the number of signals it receives from its peers in the social network. It is unclear if these same principles hold when the “pattern” by which they receive these signals vary and when potential decisions have different utilities. To investigate that, we manipulate social signal exposure in an online controlled experiment with human participants. Specifically, we change the number of signals and the pattern through which participants receive them over time. We analyze its effect through a controlled game where each participant makes a decision to select one option when presented with six choices with differing utilities, with one choice having the most utility. We avoided network effects by holding the neighborhood network of the users constant. Over multiple rounds of the game, we observe the following: (1) even in the presence of monetary risks and previously acquired knowledge of the six choices, decision-makers tend to deviate from the obvious optimal decision when their peers make similar choices, (2) when the quantity of social signals vary over time, the probability that a participant selects the decision similar to the one reflected by the social signals and therefore being responsive to social influence does not necessarily correlate proportionally to the absolute quantity of signals and (3) an early subjugation to higher quantity of peer social signals turned out to be a more effective strategy of social influence when aggregated over the rounds.
While social media enables users and organizations to obtain useful information about technology like software and security feature usage, it can also allow an adversary to exploit users by obtaining information from them or influencing them towards injurious decisions. Prior research indicates that security technology choices are subject to social influence and that these decisions are often influenced by the peer decisions and number of peers in a user’s network. In this study we investigated whether peer influence dictates users’ decisions by manipulating social signals from peers in an online, controlled experiment. Human participants recruited from Amazon Mechanical Turk played a multi-round game in which they selected a security technology from among six of differing utilities. We observe that at the end of the game, a strategy to expose users to high quantity of peer signals reflecting suboptimal choices, in the later stages of the game successfully influences users to deviate from the optimal security technology. This strategy influences almost 1.5 times the number of users with respect to the strategy where users receive constant low quantity of similar peer signals in all rounds of the game.