Though seen here looking at data on a computer, Ann Speed (1463) actually spends much of her time studying humans. A cognitive psychologist by training, Ann’s work is aimed at quantifying human behaviors, an expertise highly valued — and funded — by the Department of Homeland Security’s Transportation Security Administration. (Photo by Randy Montoya)
When people think about Sandia’s impact on homeland security, they probably think about breakthrough tools and technologies such as explosive detection devices, chemical and biological countermeasures, border security, and nuclear and radiological security systems.
But, quietly, Sandia has developed a capability in human subject studies that is turning into a game-changer, too.
That’s right: human subject studies. That line of work, says Ann Speed (1463), includes disciplines such as cognitive psychology and cognitive neuroscience that might seem foreign to the Labs’ engineering-rich landscape and culture. There are two cognition departments at Sandia, 1462 and 1463, that together have 22 technical staff members with expertise in cognitive psychology, neuroscience, and computer science.
“Sometimes, we Sandians seem to want to engineer people right out of the equation,” says Ann, who earned her PhD in cognitive psychology from Louisiana State University. “But more and more around the Labs, people are starting to realize that the human element can be just as important as the hardware, software, or engineering.”
TSA projects focus on supervisor pressures, image resolution
Ann’s Sandia work has largely been funded since 2009 by the Department of Homeland Security’s Transportation Security Administration (TSA), with other funding coming from DHS’s Science and Technology (S&T) directorate.
In the 2009-2010 timeframe, Ann’s now-colleague Andrew Cox (8116) spearheaded a multi-screener experiment that analyzed the effects of transportation security officers (TSOs) getting advice from expert TSOs on the threat status of carry-on bags. Ann led the experiment and was subsequently tapped for a follow-up project that independently manipulated supervisor emphasis on either accuracy or throughput as well as image resolution of screened baggage, and the impact of each on TSOs’ decision-making. The study focused on the question of whether higher resolution reduces the effects of supervisor pressures on TSOs by helping them in the certainty of their decisions.
More specifically, the study asked how such pressures influence a TSO’s decisions. Does the supervisor stress speed and quantity of passengers and baggage screened? Or is he or she more concerned with accuracy of found threats? With image resolution, the questions are similar: How do the varying degrees of image resolution affect the decisions being made by TSOs charged with detecting threats? Does image resolution slow activity at the X-ray station? Does it improve accuracy?
The most recent project Ann and Kiran Lakkaraju (1462) completed for TSA is known as the “lane changing” study, which focused on impacts on threat detection when TSOs are asked to switch between the pre-check (indicated by TSA as TSA Pre√) and standard passenger lanes. TSA Pre√ lanes, introduced only recently by TSA, speed throughput considerably since approved TSA Pre√ passengers are not required to remove shoes or items from their carry-on bags.
“The interesting question, the one we were asked to examine, was what does it do cognitively to a TSO when he or she switches from the TSA Pre√ lane back to the standard lane, and vice versa?” says Ann. “We know that expectations have an impact on how people make decisions, and that the actual rate of target items in a sample can also impact decision making. So we designed an experiment to independently test the effects of expectations and threat rates.”
Though she can’t reveal any details from the TSA lane-changing study, Ann says a subset of mitigations for what was found are likely going to be rolled out to airports across the country. “It’s really exciting to know that your work has had that kind of impact,” she says. Her earlier work on TSA supervisor emphasis informed some of those mitigations.
Data, data, and more data
The bread and butter of human subject studies is data. As she does with all of her TSA projects, Ann used several computers loaded with software that allowed her to present about a thousand images of baggage to the TSA officers, images captured by actual Smiths Detection AT-2 X-Ray scanning machines used at airport checkpoints. The experiments, performed with between 30 and 200 TSA officers, involve statistical analyses of how effectively the officers identify prohibited items found in some of the images.
“This kind of data collection and analysis can tell us how officers are making their decisions, how accurate they are, and what the rate of false alarms is,” Ann says. “We’re capturing and analyzing their responses and decision times in different operational environments. In the end, the data inform us and our customer about the factors that impact officers’ accuracy.”
The work, Ann adds, was — and is always — reviewed and approved by Sandia’s Human Studies Board (HSB), by TSA officials, and even by the DHS privacy office.
The success of the previous work has led directly to additional TSA-funded efforts, including a current project that explores how long TSOs can look at scanned images before their performance starts to degrade due to fatigue or other factors. Another project aims to understand the attributes TSOs bring to the table prior to training that may influence their ability to perform duties other than the X-ray interrogation of bags.
“TSOs serve many purposes, each of which requires different kinds of communication skills,” says Ann. “For instance, there are duties like communicating with passengers about things to divest [laptops, liquids] and communicating with passengers in the event a pat-down is required. They also need to possess the ability to keep passengers calm and compliant while performing the tasks required by the standard operating procedure.
“This work is unusual for Sandia, but the fact is that we’re very good at quantifying human behavior,” says Ann. “Scientists have been doing this for 150 years and have learned a lot about human behavior and how to measure it. We don’t need to put a person in an MRI machine to understand how the brain is producing the behaviors it’s producing.” Instead, she says, psychologists regularly design and execute experiments that offer scientific insight into various behaviors and how they come about.
“They [the experiments] do have to be pretty clever, though,” Ann reminds us. “Humans are thinking beings who will try to outsmart one another, so we have to be careful about experiment construction and the different variables that go into them. Even the instructions we give subjects can alter the outcome of a study.”
In addition to the increased level of attention that TSA is giving to Sandia in this area, Ann says other organizations have taken notice as well. The Labs recently signed a memorandum of understanding with the Paul Allen Institute, for example, and continues to develop relationships with the Department of Defense and others.
It’s also worth noting, Ann says, that all of the Labs’ program management units (PMUs) have funded human subject studies at some level, either as LDRDs or as work-for-others. “Many decision makers across the Labs recognize the ubiquity of the human dimension to our national security missions,” she says.
An external advisory board made up of distinguished scholars, cognition scientists, and others has repeatedly acknowledged that Sandia has a differentiating capability in this area.
“There is no other place that can do what Sandia can do in the area of human decision-making in high-consequence threat scenarios,” Ann asserts. “We are it.”